Most people do not understand how wonderfully complicated the modern day operating system truly is. They do not think about all of the efforts and the many parts that go into making the whole system work. I think that it would probably truly shock people if they were ever to know what is going on in there. But if you have ever taken programming classes in any of the top colleges in the country then you know what is going on. Mostly all of the highly accredited schools have a section in computer science that is solely dedicated to the operating system. They show you what is going on inside of there and how you can make one of your own. You will get to see the many parts that make up the operating system and how they connect to one another. The more you start to understand it, you will find that you might have a part of the operating system that fascinates you more than the others. For black hat hackers this part is usually the stack and the whole memory model.
What is a memory model?
The memory model for your modern day computer is how your memory is managed when a program is loaded onto there. Yes, the ram that you buy actually does have a purpose. Most people think that the RAM that they put in their computer is to make it go faster. That is not exactly the case. What the RAM actually does is allow you to add more of program to it when it is executed by the system. When you click the mouse button for a program to start what you are actually doing is taking that program off of the hard drive and loading it into the memory of the system aka the RAM.
That program is not the only thing that is loaded onto your RAM at the time. Parts of the operating system are as well as any other program that you might have running at the time. So when you load your program onto the memory, depending on the size of the program, you are probably only going to load only the part that you need at the moment. The other parts that you may need later will be loaded up when you really need them. The parts that are loaded right now will be deallocatted if you are not using them. That is why when you are using your computer and you have too much stuff running at the same time it seems as if it is going slower. Usually that is because the RAM is stuffed and it is now using part of the hard drive as virtual memory. A bigger RAM stick is able to avoid all of these problems. That is why people seem to think that RAM makes the system go faster.
So what does this have to do with the stack of the operating system?
This section on memory involves the stack because of what the stack does. There are two sections of memory when running a program, the stack and the heap. Objects that are going to be around for a long time in your memory are stacked on the heap. Objects that are running now or are to be executed soon are placed on the stack. When you run your software, the software is placed on the stack. The parts that the CPU is supposed to run next are placed in numerical order on this stack. A hacker is able to use this to their advantage because if they are able to get into that part of the memory and change the code that is next in line to be executed on the stack then they can take over the system. They can force the program that is running to open up a hole in another section of the system. From that hole they can load and run a bigger exploit code.
So as you can see, the stack of the memory model can be very helpful to a black hat hacker. That is why a lot of black hat and white hat hackers study this portion of the operating system. They know that they can exploit it.
There has been a lot of new protection put into the modern operating system to try and stop attacks such as these. These new protections make it so that you are not able to execute code in certain areas of the memory model. But black hat hackers are finding their way around these restrictions as well. It is an ongoing battle between the good guys and the bad guys.