What actually happens when you open a webpage

What actually happens when you open a webpage. The process of loading a webpage in detail. The science behind loading a webpage.

Ignorance is bliss. However not so for developers or for business men trying to optimize their web presence.

Recently one of our very esteemed clients wanted to get their systems running very fast. They wanted high performance and excellent page loading time. So I wrote this to explain what really happens when a browser requests a webpage. This was written not purely in technical format but to help non-technical folks understand what really happens when you load up a webpage.

Note that although this was written for PHP/MySQL, almost the same happens with any other technology stack be it .NET/IIS or anything else.

Here's what my reply was:

What I wanted to point out is, if it's good enough for amazon.com, its probably good enough for us.

The 2 second timeline is a misnomer. Here’s why. I am writing this hoping to explain various things involved so you can judge and prioritize. This should help understand where to look for performance issues.

This is what happens when you open a webpage. The to-fro means your machine to server and back.

1. DNS Resolution: the URL is sent to name server to resolve to an IP address. if its cached, good. This is a network to-fro.
2. Then after resolution, the browser sends a handshake signal to the server. This is the HTTP protocol establishing a connection. Another to-fro.
3. If the connection is a secure one (HTTPS), a challenge is sent to prove that the site is authentic. Another double to-fro on network.
4. Resource Request: Now the browser requests a resource, usually a webpage. This is a request that goes from browser to server. One network to.
5. Server now looks at the requests and invokes the handler of the request. This is the PHP language processor with arguments like filename. This is spawing a process or a fork in technical terms. Sometimes this is an in-process execution like nginx. This is where RAM is hit.
6. PHP processor loads itself in memory and opens the file. Then it starts executing the file one per line. This may have other includes of other files. This is where disk is hit. Sometimes the file may be in cache and reduce the disk/loading.
7. As PHP executes depending on how the code is written it starts hitting the database one or multiple times to get the data it needs. This is where database is hit.
8. When a request comes to database, first a connection is established. This may be a new connection or an existing one from connection pool. If this is a new connection this is a rather costly process as database needs to establish security of the connection, allocate resources and may need locking at global level of the OS to do some of these things. This is another place where OS, disk, RAM and files are hit. If the database is on another server, network is hit. This takes another set of sequence to establish connection, though not as many steps since most connections are using custom protocols.
9. Now the database looks at the SQL, parses it, rearranges it to make an execution plan. This is where it needs to decide whether the data is in memory already (from previous execution) or if it can find the data from index or does it needs to perform a serial look up. If the execution happens more than 3 times within a normal period of time, the execution plan is cached. The caching depends on several factors, most importantly the SQL matching.
10. The database also needs to decide the isolation level which dictates whether the data can simply be read from the table or does it needs to lock the table completely blocking other connections. Usually it is a normal read. Note that this is for read. See write operation below. This is another place where disk, RAM and CPU are hit.
11. If the database needs to load the data, it needs to open the data from files and load in memory. If it is not properly indexed, it needs to look at each record to find the relevant data which is a HUGE performance hit. This is a disk hit.
12. Now the database executes the query, performs join, filters, grouping etc, collects the data and returns the data through connection. Note that it does not return all data. Only 1-10 records are returned depending on the database client configuration for a parameter called prefetch count. The rest is temporarily stored in memory or if too large, temporary tables. Different databases handle this differently. Some databases may not even fetch all the data unless ordering or aggregation is involved. This is a CPU, RAM intensive operation.
13. Now the PHP takes the data row by row, fetching more as it iterates over the rows and concatenating strings in the process to combine data thereby forming the HTML line by line. This is a CPU/RAM operation.
14. Once the HTML is compiled completely it is sent back to the web server (apache or nginx). At this point the PHP interpreter releases memory and closes the process.
15. Now the web server returns the data/HTML to the browser. This is a network return operation.
16. At this point we have only received the first return (HTML) of the page. Now the browser reads the HTML line by line and if it needs any files (usually images, CSS, JS) it starts requesting these one by one. Each of these is a complete request like the one we are discussing. The initial DNS lookup and handshake is not needed. Each file request requires a) Network reach to server, server looking up the file, loading from disk or cache, returning the file.
17. As soon as a JS file is hit in browser, it stops other fetches and first reads that, executes that in context and then resumes fetching of files. Hence it is strongly advised to load JS files at end and reduce the number.
18. Once all the HTML is fully parsed, the browser starts rendering the content, applying the CSS rules and displaying the page. This is a client side operation.

Phew. Just so you know I skipped the actual technical details so I don’t really make you go crazy. For any performance realizing these are very important.

Now the answer to your 2 second timeline. Overall each request has: 1) Client browser 2) Network from your machine -> nearest access point/router > relay > provider > city provider > state/national provider > hosting providers network > reverse of state/city/nearest > hosting company > firewall > switch > server. 3) Activities of server (see above) 4) Return trip from server to your machine (for each HTML/CSS/JS file).

Also note that each request of TCP/IP (HTTP) is broken into multiple packets. Packets can take any route to reach destination. So a 1 MB page may be divided into 20 packets and 5 reach through direct route whereas 15 may get routed from Canada to UK to Singapore to Japan to LA to Canada. Most of the time it is not so bad, but the journey described does vary at local level (different relays in different neighborhoods of your city). Hence the predictability of the network to give a consistent 2 second speed is not entirely dependent on how well you optimize the server.

Enjoy!