Need help from experienced ASP.NET developers.

Witte

Marauder
Joined
May 21, 2005
Posts
5,713
Society
Delta Force Elite
Avatar Name
Vlugge Witte Harrie
I just got a mail from my host that there are processes that consume all of the processor time on the IIS and/or MySQL server. The site (entropedia) is running for quiet a while now, and I have totally no clue what causes this all in a sudden. My host gave me information which requests caused it, but I cant find any flaws in the code.

For now they have shut down the site, untill I fix it. So any help to solve this is welcome. I can send the sourcecode on request.
 
where is the server host based and how many time/cost would it take to get it put onto a dedicated server? would still be good to clean up dodgy code, but at least it wouldnt get cut off.
 
I'm more of a linux/apache kind of guy... but if you want to send me any relevant code, I'd take a look for sure. You can send it to carebear.doah (at) gmail (dot) com

Good luck
 
Wouldn't call myself an expert, but have definitely done a lot of ASP.NET coding and would be happy to take a look at it. I'll send email address by PM.
 
where is the server host based and how many time/cost would it take to get it put onto a dedicated server? would still be good to clean up dodgy code, but at least it wouldnt get cut off.

I have looked for several hosts, but dedicated servers are too costy for me. This is only an option when someone can offer this for free. The current host is www.dailyrazor.com
 
You can..

A. fork up 10$ extra to run a shared virtual server (eg, no CPU usage limit)
B. switch to a better hosting company that WON'T shut down your sites
C. use php/mysql next time :ahh:
 
Witte has inherited the system with asp, so thats where we are... although if theres problems is a rewrite off the cards? how difficult is that, given the backend is in Mysql?

i am looking at a dedicated server for a project which would have space, but asp is no good unless anyone is familiar with running asp on linux.
 
I got the suggestion from Carebear that it might be caused by the fact that each picture that is loaded opens a connection to the database. That might overload the database. Although I am not sure how that can result in a process being open for a long time consuming all processor time.

If there are any more experts out there, your help is welcome ;).
 
Been looking at the code, and it looks good at first glance -- work keeps interrupting though :D

I'll take a look at it again later tonight and let you know if I find anything - my C# is a bit rusty though..
 
I got the suggestion from Carebear that it might be caused by the fact that each picture that is loaded opens a connection to the database. That might overload the database. Although I am not sure how that can result in a process being open for a long time consuming all processor time.

If there are any more experts out there, your help is welcome ;).

Well, you could turn off the images temporarily to see if that is indeed the problem. ie... just don't paint anything in the Image render method. It would be better than the whole site being down anyway.

I didn't see from the email you sent where it specified that a particular process was taking a long time though. It just said generically "Your website application is overloading IIS server:" and showed you the current running processes. Maybe I missed it though.

CareBear
 
One other thing I though of as a possiblity.... are you sure the problem doesn't exist within the database itself?

Disclaimer: I have absolutely 0 MySQL experience.

Reason I'm asking is that if say for example an index was missing, data was fragmanted, or files/disks were running out of space, then the database may not be performing optimally. If the database is slow than any request to the database will also be slower.

Just a thought....
 
Well, you could turn off the images temporarily to see if that is indeed the problem. ie... just don't paint anything in the Image render method. It would be better than the whole site being down anyway.

I didn't see from the email you sent where it specified that a particular process was taking a long time though. It just said generically "Your website application is overloading IIS server:" and showed you the current running processes. Maybe I missed it though.

CareBear


It said so in a different email, after I asked them to be more specific:

Well, all of a sudden your application began to consume too much of server processor time and disk operations through IIS server and MySQL requests. We showed you 5 connections to your website that were running for more than 4000secs giving high procesor load.

The odd thing is, that this problem came all in a sudden just out of the blue.
 
One other thing I though of as a possiblity.... are you sure the problem doesn't exist within the database itself?

Disclaimer: I have absolutely 0 MySQL experience.

Reason I'm asking is that if say for example an index was missing, data was fragmanted, or files/disks were running out of space, then the database may not be performing optimally. If the database is slow than any request to the database will also be slower.

Just a thought....

It could be but how can I help that? Isnt that the task of the host to fix? Might be they are just not that good ofcourse.
 
It could be but how can I help that? Isnt that the task of the host to fix? Might be they are just not that good ofcourse.



That certainly depends on the agreement you have with the host. My guess is they are responsible for running the server, but that you (or your predecessor) created the database it is running in. The host would then be responsible for the database application, but you would be responsible for the maintenance of the objects within the database (tables, indexes, etc.)

I come from a SQL Server background and there are plenty of commands and tools for checking the performance and validity of database objects - not sure what exists for MySQL, especially in a hosted situation. Perhaps ask your host if they have any tools available.
 
Well, all of a sudden your application began to consume too much of server processor time and disk operations through IIS server and MySQL requests. We showed you 5 connections to your website that were running for more than 4000secs giving high procesor load.

This look to me to a non ending loop.
Can be caused by looking after som data that does not exist and not use any way to stop the process.
Or som recursive process with som missing flag , or that kind of things.

I dont think the problem come from the database itself , as mysql would return an eror , or just slow down the whole database.

I am not specialist about asp , i dont it at all , i am more php.
But in php , all process are stoped after 30 seconde by default.
I dont understand how a process on a web server can run for 4000 sec...
 
It said so in a different email, after I asked them to be more specific:



The odd thing is, that this problem came all in a sudden just out of the blue.

I don't remember... were images uploaded by users on your site? If so... you might want to check some of the recent changes in that area, maybe a max length of the data for the images. It seems like a long shot, but maybe someone upload a ridiculously sized image. :dunno:
 
I am not specialist about asp , i dont it at all , i am more php.
But in php , all process are stoped after 30 seconde by default.
I dont understand how a process on a web server can run for 4000 sec...

The setting is changeable in php via set_time_limit, maybe there is a setting in asp that can be lowered to prevent long running requests.

*edit*

Looks like the setting is in the web.config file located either in the same directory as your source, or one of its parent directories...

It will look something like...

Code:
<configuration>
  <system.web>
  <httpRuntime maxRequestLength="4000"
    enable = "True"
    requestLengthDiskThreshold="512
    useFullyQualifiedRedirectUrl="True"
    executionTimeout="45"
    versionHeader="1.1.4128"/>
  </system.web>
</configuration>

I wonder what your executionTimeout is set to... the default looks like it should be 110 seconds if none is set.
 
Last edited:
The setting is changeable in php via set_time_limit, maybe there is a setting in asp that can be lowered to prevent long running requests.

Yes i know , but you can use that "variable" only if its alowed in the init file , and usual , host dont alow it.
You have that right if you ask kindly :)
And its not the kind of things you change "for fun"... but better be sure your code is good , or you can make you site very slow for several time...

I do think that in most of case , 30 sec is way enought...
And i dont see any things in entropia that would need long time to execute.

Maybe trying to do it other way.
There is maybe a setting in ASP that alow you to limit time. would be nice to use it , and stop any process after 30 sec.
There maybe , see what is stoped , shen and why...
 
Oh this is so sad.

I do hope you guys can get this sorted out :)

As for why it has happened suddenly, the only thing that has changed in the last week is all these new armour pics etc. This ties in with the image thing. Could this be a cause??
 
Oh this is so sad.

I do hope you guys can get this sorted out :)

As for why it has happened suddenly, the only thing that has changed in the last week is all these new armour pics etc. This ties in with the image thing. Could this be a cause??
Maybe but easy to know if image are keep in jpg for exemple.
just sort file by size... the over sized image can be find very fast.And then , include an image veryfication code " format , size...).
If image are stored in database , a little code can be write to find the oversized entry too.

But again , usual , when you write code for "public" use as upload , there is always a format and size chek.
 
I don't remember... were images uploaded by users on your site? If so... you might want to check some of the recent changes in that area, maybe a max length of the data for the images. It seems like a long shot, but maybe someone upload a ridiculously sized image. :dunno:

Shorter shot:

Did you try to scan the new pictures (i guess they are .jpg) ?

Asking because you are using .ASP - which is running on a platform from a software company which is known to have some serious issues with buffer overflows in their libraries to handle image files.

Tussi
 
I got the suggestion from Carebear that it might be caused by the fact that each picture that is loaded opens a connection to the database.
That should be rewritten quickly. It's no good idea to open a database connection for every image loaded.
Sadly you're not using a *nix server because then you could do some very nice tricks with the filesystem (eg. id to filename conversion w/o any database lookups).
That might overload the database.
The database itself, or other stuff in the way (depending on how you connect to the DB?).
Although I am not sure how that can result in a process being open for a long time consuming all processor time.
Strange things can happen the moment you overflow some table limits (eg. hitting a hard maximum connections limit on client or server side).

Tussi
 
The picturesize is checked upon upload, and is handled by "gallery.aspx.cs". It seems the system load problems came from image.aspx. It doesnt do that much except lookup an image in the database. When the image doesn not exist it return an error. I dont understand how this code can take 4000 sec to execute while causing a 100% load.

A copy of the code:

Code:
	public partial class LoadImage : PageBase
	{
        private static ImageCodecInfo GetEncoderInfo(String mimeType)
        {
            int j;
            ImageCodecInfo[] encoders;
            encoders = ImageCodecInfo.GetImageEncoders();
            for (j = 0; j < encoders.Length; ++j)
            {
                if (encoders[j].MimeType == mimeType)
                    return encoders[j];
            }
            return null;
        }

		protected override void Render(HtmlTextWriter writer)
		{
//			System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.InvariantCulture;
//			System.Threading.Thread.CurrentThread.CurrentUICulture = System.Globalization.CultureInfo.InvariantCulture;
			string id = Request.QueryString["id"];
			if(id == null)
			{
				id = "0";
//				Response.StatusCode = 404;
//				return;
			}

            bool micro = Request.QueryString["Micro"] == "1";
            bool micro2 = Request.QueryString["Micro2"] == "1";
            bool small = Request.QueryString["Small"] == "1";
            bool normal = Request.QueryString["Normal"] == "1";
            bool original = Request.QueryString["Original"] == "1";

			string type;
			if(micro) type = "Micro";
            else if (micro2) type = "Micro2";
            else if (small) type = "Small";
			else type = "Normal";

			MySqlCommand cmd = new MySqlCommand("SELECT " + type + ", Gallery.Type, ChangeLog.Time FROM Gallery LEFT JOIN ChangeLog ON ChangeLog.Item=Gallery.ID AND ChangeLog.Chart='Gallery' WHERE Gallery.ID=?ID", Con);
			cmd.Parameters.Add("?ID", id);
            using (MySqlDataReader reader = cmd.ExecuteReader())
            {
                if (reader.Read())
                {
                    byte[] data = (byte[])reader[0];
                    if (normal)
                    {
                        MemoryStream stream = new MemoryStream(data);
                        using (DrawImage img = DrawImage.FromStream(stream))
                        {
                            int width = img.Width;
                            int height = img.Height;
                            if (width >= height)
                            {
                                if (width > 401)
                                {
                                    height = (int)((float)height * (float)((float)401 / (float)width));
                                    width = 401;
                                }
                            }
                            else
                            {
                                if (height > 401)
                                {
                                    width = (int)((float)width * (float)((float)401 / (float)height));
                                    height = 401;
                                }
                            }
                            Bitmap bmp = new Bitmap(width-1, height-1);

                            Graphics g = Graphics.FromImage(bmp);
                            g.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic;
                            g.FillRectangle(Brushes.White, 0, 0, bmp.Width, bmp.Height);
                            g.DrawImage(img, 0,0, bmp.Width, bmp.Height);
                            MemoryStream mstream = new MemoryStream();


                            ImageCodecInfo jpegEncoder = GetEncoderInfo("image/jpeg");
                            EncoderParameters parms = new EncoderParameters(1);
                            parms.Param[0] = new EncoderParameter(Encoder.Quality, 80L);

                            bmp.Save(mstream, jpegEncoder, parms);
                            data = new byte[(int)mstream.Length];
                            data = mstream.GetBuffer();
                        }

                    }

                    Response.ContentType = micro || small ? "image/jpeg" : (string)reader[1];
                    if (reader[2] is DateTime && !original) Response.Cache.SetLastModified((DateTime)reader[2]-(DateTime.UtcNow - DateTime.Now));
                    Response.OutputStream.Write(data, 0, data.Length);
                }
                else
                    Response.StatusCode = 404;
            }
		}
	}
 
That should be rewritten quickly. It's no good idea to open a database connection for every image loaded.
Sadly you're not using a *nix server because then you could do some very nice tricks with the filesystem (eg. id to filename conversion w/o any database lookups).

I am not even sure how this can be coded that way.
That would mean the image are stored in data base , and you use an other database , to reade the image name , then open a new connection to image data to read one image...
Very strange coding...
And usual , conection to database are handled way better by system it self.
Keeping connection open and use again same , or auto close after use... dpened on setting and what you do.
 
That should be rewritten quickly. It's no good idea to open a database connection for every image loaded.
Sadly you're not using a *nix server because then you could do some very nice tricks with the filesystem (eg. id to filename conversion w/o any database lookups).

The database itself, or other stuff in the way (depending on how you connect to the DB?).

Strange things can happen the moment you overflow some table limits (eg. hitting a hard maximum connections limit on client or server side).

Tussi

My mistake, it uses the same connection for all pics that are loaded. I mean to say that each image is loaded from the database each time its requested. (unless its cached clientside)
 
Strange things can happen the moment you overflow some table limits (eg. hitting a hard maximum connections limit on client or server side).

Tussi

When I run a lot of queries in a short time using one connection, is that also a bad thing? If so, would adding a small delay between each query solve it?
 
It's been a time, and what little dev I do these days is much more on the DB side of things... and I am a little drunk. So I am liable to ramble, and will quite probably be talking rubbish, but here goes :D


Firstly it looks like you are opening the connection elsewhere and keeping it open...

MySqlCommand cmd = new MySqlCommand("SELECT " + type + ", Gallery.Type, ChangeLog.Time FROM Gallery LEFT JOIN ChangeLog ON ChangeLog.Item=Gallery.ID AND ChangeLog.Chart='Gallery' WHERE Gallery.ID=?ID", Con);

or simplified

MySqlCommand cmd = new MySqlCommand("<sql stuff>", Con);

Con is your SQL connection and was presumably opened elsewhere with something like

MysqlConnection con = new MySqlConnection(<Connection String>);
and presumably
con.Open();

This is a dodgy way of doing things, and it would be much better to open the SQL connection just before executing the command
Con.Open();
using (MySqlDataReader reader = cmd.ExecuteReader())


and then close it at the end with
Con.Close();

This would free up the connections when they are no longer needed, instead of waiting for garbage collection to get them. The downside of this, is that you would need to check where else that con object is used and change that all that code similarly. (On a really really petty point, for which I apologies... it is normally called Conn not Con!)

One thing you could do easily is to close your datareader and two memory streams by changing your code to something like:


data = mstream.GetBuffer();
mstream.Close();
}
stream.Close();

}

Response.ContentType = micro || small ? "image/jpeg" : (string)reader[1];
if (reader[2] is DateTime && !original) Response.Cache.SetLastModified((DateTime)reader[2]-(DateTime.UtcNow - DateTime.Now));
Response.OutputStream.Write(data, 0, data.Length);
}
else
Response.StatusCode = 404;
}
}

reader.Close();
}


That should free up some memory I guess, but don't know how much :dunno:


The other thing which is also worth looking at is the DB connection pooling. If you look at your connection string for Con see if it has Pooling = False or Pooling = No If it does change it, and if it doesn't add it so you have Pooling = True

That should force connection pooling on if it isn't already. Causing the connections to be reused, instead of spawning loads of new DB connections. (For a simple query opening the connection can often take more resource and time that running the query itself.)

As I mentioned earlier the Con connection must be opened elsewhere, it is also worth checking if it closed elsewhere and in particular if you have a couple of line like:
Con.Close();
Con.Dispose();


Closing is good and we definitely want that, but disposing of it can under some circumtances mess up connection pooling, by stoping the pool manager from recognising connections with the same ID. Worth a look.


As to why you've suddenly got this problem? Might be worth asking your host if they recreated any ODBC connections if you connect to MYsql via one? Have they upgraded any .Net, ADO, MySQl recently? Applied any MS updates? etc... I'm assuming not as I'm sure they would have said, but it is worth asking the question.

Or could it be as someone else suggested all the new VU pictures, causing a DB table to suddenly grow?

If it is a DB performance issue I can probably be a lot more help than I have been with the code, but you need to get me the DB schema somehow.

Bla! enough witterings from me...

Peatie
 
The picturesize is checked upon upload, and is handled by "gallery.aspx.cs". It seems the system load problems came from image.aspx. It doesnt do that much except lookup an image in the database. When the image doesn not exist it return an error. I dont understand how this code can take 4000 sec to execute while causing a 100% load.

A copy of the code:
<snip>
Should i get that right then the code will load the image from the DB, then every time a picture with normal size is requested resize the image with HighQualityBicubic, then recompress it as jpeg (and not clean up the buffers used behind it).

So i guess that the moment some users are coming simultaneously then it could take a moment to fullfill their requests. Should it be more users then the script will allocate quite some memory (because it needs to copy the image from one buffer to another), so should there be enough requests then the system will run out of real memory -> start to page to get more virtual memory -> will get even slower -> users will press F5 (because they don't see the picture and get impatient) -> death-spiral.

I am not even sure how this can be coded that way.
That would mean the image are stored in data base , and you use an other database , to reade the image name , then open a new connection to image data to read one image...
Very strange coding...
And usual , conection to database are handled way better by system it self.
Keeping connection open and use again same , or auto close after use... dpened on setting and what you do.
As i grasp the code posted the image currently is stored in the database.

Let me try again with an example:
Normal way to handle such stuff would be (imho) to store the original uploaded image in the filesystem (which is optimized for such stuff - eg. gives nice caching and features like direct file to tcp stream copy) and create resized versions once on demand (first access).

My plan for dealing with a lot of resized images would be to store them inside the filesystem with the following pattern:
Code:
/images/original/<file>
/images/<sizename>/<file>
and then put an error handler (.htaccess on 404 calling a script) onto this directory which will only be raised in case that a file not found (404) - so script then can either create the resized version (and store it in the corresponding folder for the next request) and then return it (so that this initial request won't fail), or throw an error in case the original isn't found or the size requested isn't defined.

In cases that the file exists there would be no load whatsoever on the DB because the webserver would directly pipe it from the filesystem to the client (and stuff like OS level caching can work at its best).

And in cases that you want to delete an image you just would do
Code:
rm /images/*/<file>
should you want to update an image then you would run the delete before you write the new version into /images/originals/. In case you chance a size definition you simply would do
Code:
rm /images/<sizename>/*
and the corresponding resized images would be rebuild on next access.

So the images would be converted into a specific size only once (not on every access) and then served without load to the DB and without script overhead.

Hope that clears it a bit

My mistake, it uses the same connection for all pics that are loaded. I mean to say that each image is loaded from the database each time its requested. (unless its cached clientside)
No good plan (IMHO).

OK, the DB might cache the row in case it's used recently, but the filesystem cache is way more effective for stuff like this (more direct) without the need to let the script grab it from the DB, mangle it (which it seems it does) and then hand it to the webserver to send it out.

I hope this is helpfull in some way.

Tussi
 
Last edited:
Maybe but easy to know if image are keep in jpg for exemple.
just sort file by size... the over sized image can be find very fast.And then , include an image veryfication code " format , size...).
If image are stored in database , a little code can be write to find the oversized entry too.

But again , usual , when you write code for "public" use as upload , there is always a format and size chek.

There must be a check on the jpeg header sanity on upload or the system would be wide open to a decompression bomb attack. Should there be no such check then a malicious user would be able to send it a jpeg which is 10KB in size but will decompress to a 100000 x 100000 pixel bitmap - and blow your system the moment you try to resize it with the code posted.

And with the file system tricks i ment using symlinks (or hardlinks, but then you would need better sanity checking on update/delete) to convert from ID to filename inside the filesystem without the need to start a script in the first place.

Tussi
 

Thanks for the suggestions. As it works now, a connection is opened on page load in the pagebase class. This connection is used untill the page is ready. Opening a new connection for every query seems uneffecient to me.

open:

Code:
        public PageBase()
        {
            if (Context != null)
            {
                Con = Service.GetConnection();
            }
            WikiSyntax = new WikiSyntax(this);
        }

Code:
        protected override void OnUnload(EventArgs e)
        {
            base.OnUnload(e);
            Service.CloseConnection(Con);
        }

The site uses its own connectionpool, so I deliberatly turned off MySQL connection pooling. Using the method you suggested would mean I need to rewrite a large part of the site. Since I am not sure if this causes the problem, I am not sure if its worth the efford.

About the buffers, I think that might be an issue.
 
Last edited:
Back
Top