Data warehouse

hello I’ve been using one of the other free hosts for Data warehousing but after 2 years they suspended my account .
so here is my application : i have 2 fully automated servers which send request to web-server every second .and web-server updates the database .
is my application allowed to be deployed in InfinityFree servers ?
since i didn’t notice anything about Data warehousing in terms and service , i am posting this here !

thank you !

Thank you for asking about this beforehand, I’m glad you did.

The short answer to your question is: no, we cannot host such applications.

We provide a website hosting service with PHP and MySQL. We do provide databases for you to use, but these are intended for small datasets and light, continuous usage (like loading blog posts and comments when a web page is opened). Data warehouses typically involve large data sets and complex aggregations, usually in short bursts, which put a lot of stress on the database server. A database for website hosting is simply not intended for this. While we don’t specifically talk about data warehousing in our terms, we do have a clause about overloading the servers.

Our abuse detection system would probably shut down your account if you try to hammer is with complex processing queries, and I would image that that’s what happened on your previous hosting provider as well.

There is also the fact that we only provide hosting for websites to be viewed in browsers (which is also covered in our terms). We have a security system in place which prevents most kinds of automated access to your website, including your automated submission servers. So the script submitting the data would just see errors.

That is, if you can upload the code to begin with. Scripts which automatically collect and store all input are often associated with malicious purposes, like keyloggers and phishing scripts. So your account may already be shut down before you’ve even finished uploading your code.

If I can make a recommendation, I would suggest to look beyond web hosting services to provide your data warehouse. Instead, you could consider to use more purpose built data processing or IoT data collection services. I know for example that Google Firebase has quite scalable databases with a free tier. But I’m not exactly an expert on these services.

2 Likes

I understand your concern about server overloading , but my application is not designed to overload the servers , 2 requests every second is not that much to overload the webserver ! . There are websites with multiple simultaneous click/request .
Anyways, i would like to know if i can deploy my application in an upgraded account !!?

i would also know , to what extend i should increase the 1 sec time span between each requests to to avoid suspending my free account ?
like if i make it 10 sec is it ok ?

True, there are websites with more than two requests in a second. But most websites don’t have 2 requests per second in every second of every hour of every day.

Come to think of it, there are 86,400 seconds in a day, so your website would receive 172,800 requests per day. We enforce a limit of 50,000 hits per day. So that kind of traffic would get your account suspended for high traffic, even if there was no PHP code or database activity at all.

But there is PHP code and database activity in your application. And quite a lot in fact.

Judging by your original message, am I correct to assume that you’re storing some kind of measurements from these two servers? Because then your application would collect 172,800 data samples per day as well. And, assuming one measurement per request, that would result in at least 172,800 database rows per day, or over 5 million per month. Depending on how you do data retention and downsampling, your database would grow very big very quickly. It would quickly go way beyond any measure of fair use for a website hosting service.

And that’s only inserts, which are fairly cheap. But given that you call it a “data warehouse”, I assume that you intend to do analysis on these measurements as well. And doing complex analysis on a large dataset usually involves very big and very costly queries, which would probably cause high load on the database.

You’re right, two requests doesn’t sound like that much. But that’s the tiny tip of a massive iceberg. In my experience, “data warehouse” is impossible without “database overloading”. Which is why most companies use a separate database for their warehouse, or even a purpose built data warehousing engine.

You’ll have to check this yourself with iFastNet, who provide the premium hosting. My guess would be that the answer is “no” because the same terms of service apply (including the “only websites” clause), even if they are not enforced as strictly.

I think you missed the point of my previous message.

If you read all my concerns, you should note that I listed two reasons why you would not be able to send a single request to your database hosted here. One of them is a guaranteed show stopper, which basically makes all the discussion about database overloading and request rates irrelevant.

Maybe I should have been more clear about that up front. But I also wanted to let you know why your “just two requests” application can easily generate a massive data set which is way beyond what’s acceptable for any website hosting service.

3 Likes

No , I’m not doing any analysis with the gathered data I’m just generating constant report , 2 servers collect information (temperature , uptime , TH , and current date and time ) of 2 mining machines , so there will be information of 4 mining machines displayed in the main website page , I’m not storing these informations , I’m just replacing new stat of machines with older stat at certain time span , so there won’t be million rows in database , there will be only 4 rows .
Currently I’m not even using database for this , I’m just using text file to store current stats and with the use of file_put_contents() function i replace the newer value with older one .

If you are sure about limit of 50000 hit per day . Then I’m going to increase the gap between each requests to 1 minute , in that. Case I’m gonna have only 2866 hit per day which is much lower than 50000 cap . Is this doable ?

You still seem to miss the key points of my message. There are two key reasons why you cannot upload a single measurement to your website here. A malicious code scanner may stop you from uploading your application and a browser validation system can prevent you from pushing the metrics to your application.

I understand how the system works now. But for future communication to us and to other companies, please don’t call it a “data warehouse”. To quote the Wikipedia article about what a data warehouse is:

They store current and historical data in one single place that are used for creating analytical reports for workers throughout the enterprise.

You don’t store historical data and don’t do analytics. What you’re doing is not data warehousing, it’s remote monitoring.

Come to think of it, wouldn’t a service like Datadog be a much better fit? Or maybe just pushing the metrics to a log aggregator like LogDNA?

Your setup sounds like an awfully inefficient way to read the current status from the mining machines. I take it there really was no way to just read the status of these servers on demand?

But if you’re not storing the data, I don’t see any reason why this wouldn’t work on premium hosting. But again, you’ll have to check with iFastNet to be sure.

2 Likes

Yea you’re right , my mistake !
I will definitely check those you have mentioned !
About checking status by demand : these machines are having inbuilt webserver with web based control panel exactly like a router . And since they are running on dynamic ip , i can access them out of their local network and it is not convenient to check 4 machines each time . So with the help of 2 old Android phones I’m collecting specific data from these machines 2 by 2 located in different cities and displaying in all in one platform for ease .
About browsers verification system , i must say I’m using pretty powerful Android app called Automate , which is doing this automation perfectly and can request exactly like a browser .

I was running this setup for about 2 years on other hosting website and i didn’t have any problem with this .

Thanks for your time .

1 Like