Streams in C#

Before your programs know and understand the content of the information that's being passed over the network, they need a reliable way to transfer and receive information.

Streams offer just that. But before diving too deep in the network side, we need to take a look at streams in general! They are one of the fundamental structures of interacting with data outside of your program (not just through networks)!

When I started out as a developer, they used to confuse me a lot, but hopefully in this post I can help you leave the confusion behind!

The Base Stream Class

When talking about streams, we're strictly talking about transferring data; and when your only concern is just moving data, that means that the data is raw bytes. The base Stream class in C# represents that idea wholeheartedly.
The base class only provides direct access to an ordered sequence of bytes, but it doesn't give a meaningful way of understanding the data.

You can think of it as like the Highway over which the data passes through. It's the road, you don't care about what types of cars go through it.

A stream is an active connection to the data source. Like any data source it needs to be opened and closed and then disposed before you're done with it. This should probably make sense to you if you've already worked with external connections such as database connections and such.

External connections mean that our program must interact with the underlying Operating System in order to get what it needs, that also makes our program responsible for the resources the OS is giving it, thus why we need to open, close and dispose of such connections.

The Stream class gives us three important actions: Read, Write and Seek an index of a sequence of bytes.

Writing to Streams

The streams do not bother themselves with how complicated objects are represented. It's simply not their job, they are the road over which raw data travels. so if we want to send data over streams, we first have to convert/serialize our data to bytes (byte[], or byte depending on the size of the data).

var m = "stream data into a file";
var bytes = Encoding.UTF8.GetBytes(m);  

// Notice the ioStream's type => Stream
using Stream ioStream = new FileStream(@"strm.txt",   
    FileMode.OpenOrCreate);  

if (ioStream.CanWrite)  
    ioStream.Write(bytes, 0, bytes.Length); 

else  
    Console.WriteLine("Couldn't write to our data stream.");

Notice the ioStream variables type, it's a Stream not a FileStream. That's because it really doesn't matter where the data is coming from. If we change the FileStream to a NetworkStream, how we work with that stream would mostly be the same.
The Stream class provides us with enough data, that includes letting us knowing if the stream is ready to work with with the CanWrite property. This also, in a small way, helps error handling by avoiding try/catch blocks.

Since streams work exclusively with raw data, they don't bother themselves with anything else. We must provide them with raw data, so no matter what the data is, we must serialize it into a sequence of bytes.

We can convert strings to bytes using Encoding.UTF8.GetBytes method.

In the same way the a stream has no idea of the actual contents of the sequence of data its transferring, it also doesn't know where to start reading and when to stop. it needs the array of bytes, then instructions on where to start reading from, and precisely how many of those bytes it should write.

The second parameter in the Write method signature is your starting index. It starts at 0, just like the array does. The third parameter is the total number of bytes you want to send in this Write operation.
There is a runtime error checking on this and if you try to send more bytes than there are left in the array (starting from whatever index you designate), you'll get an index out-of-bounds error.

You're giving the stream the data, and where to read from that data.

Seek

So far we know how to write to a stream. If you run the previous code multiple times you find yourself a file that doesn't seem to change at all!
In reality, the file is changing. Every time you execute your code, it's opening a stream to the file, and writing to it starting from the files 0 index until the end of the given byte array. But we often need to also tell the stream where to start writing or reading data.

Every time we open a stream connected to a data source, we're getting the complete ordered list of bytes stored at that source, along with a pointer to the start of that array. Every operation we execute on the stream moves our pointer in one direction. If we write 10 bytes, we find ourselves 10 positions further down the array than when we started. So, each of our primary operators(Read/Write) can only ever move in one direction, forward, from the current stream index when we start executing them.

for example, if we write to the stream twice:

var m = "stream data into a file-";
var bytes = Encoding.UTF8.GetBytes(m);

using Stream ioStream = new FileStream("strm.txt",
    FileMode.OpenOrCreate);

ioStream.Write(bytes, 0, bytes.Length);
ioStream.Write(bytes, 0, bytes.Length);

The result would be the same string twice in a row:
stream data into a file-stream data into a file-

We can make the stream read or write from a certain position in the sequence using the Seek method on the Stream instance. This makes the stream to seek that index, relative to the given indexes.

The SeekOrigin enum gives us the three useful origins we might want to use in most cases: Begin, Current, End.

For example in our previous code, if want to stop overwriting the message every time, and actually append a string to the end of the file every time the program is executed, we can simply make the stream fetch the end of the file and start writing to it from there like so:

var m = "stream data into a file-";
var bytes = Encoding.UTF8.GetBytes(m);  

using Stream ioStream = new FileStream("strm.txt",  
    FileMode.OpenOrCreate);  

if (ioStream.CanWrite) 
{
    // set the pointer to the end of the file
    ioStream.Seek(0, SeekOrigin.End); 

    // now write to the stream from the end
    ioStream.Write(bytes, 0, bytes.Length);  
} 

We're telling the stream to first go to the end of the stream, and start it's operation from there:

Reading from streams

Each time the read operation is executed the streams cursor moves forward by one index. Data is read one byte at a time. You have to store the data you read in an array, or a single byte depending on your usage, and you have to make sure the array exists before you read.

var m = "stream data into a file";  
var bytes = Encoding.UTF8.GetBytes(m);  

using Stream ioStream = new FileStream("strm.txt",  FileMode.OpenOrCreate);  

if (ioStream.CanWrite)
{  
    ioStream.Seek(0, SeekOrigin.End);  
    ioStream.Write(bytes, 0, bytes.Length);  
}  
else  
    Console.WriteLine("Couldn't write to our data stream.");  

if (ioStream.CanRead)  
{  
    byte[] destArray = new byte[10];  
    ioStream.Read(destArray, 0, 10);  
    string result = Encoding.UTF8.GetString(destArray);  
    Console.WriteLine(result);  
}

While this approach works, there are limitations to it that make the code not-so-clean to work with. There's the issue of using the old-style arrays and their fixed size. It forces you to specify the size of the array, which in hand makes you probably use a reasonable maximum size for the array.

Stepping up from Stream

While the base Stream class opens up a lot of opportunities by giving us meaningful yet simple ways to manipulate streams, a lot of the work is repetitive and tedious most of the times. This is where we move on to more powerful types which ease each process of reading and writing.

JSON

before moving to general readers and writers, it's worth mentioning the most famous specification for data transfer around the web, JSON.
In .NET we have the famous Newtonsoft.Json which is easy and powerful to work with.
Since .NET Core 3.0, Microsoft has developed an alternative, System.Test.Json. the focus of this new JSON library is on performance and fine-grained control over the serialization process. As a result, the feature set of the System.Text.Json library is severely limited when compared to Newtonsoft.Json.

StreamReader & StreamWriter

Since most of our work consists of using strings to capsulate complex data with json, it makes sense to have types tailored for working with strings.
The StreamReader & StreamWriter classes sub-class the TextReader class from the System.IO namespace, and extend its functionality to interface directly with byte streams.

Even though we're using a FileStream here for demonstration purposes, for real network programming, we'll want to connect directly to a data stream to a remote resource.

When working with strings from remote resources, knowing the specific encoding with which to translate the incoming bytes is key. If a character is encoded as UTF32, and decoded using ASCII, the result wouldn't match the input, rendering your output string a garbled mess. If you ever find a message that you've parsed to be indecipherable, make sure you're using the right encoding.

Using these classes will reduce the amount of work you need to do when working strings to just a few simple lines (if you know what you're doing!):

ComplexModel testModel = new ComplexModel();
string message = JsonConvert.SerializeObject(testModel);

using Stream ioStream = new FileStream("strm.txt", FileMode.OpenOrCreate);
using StreamWriter sw = new StreamWriter(ioStream);

sw.Write(message);

Since these classes are designed to work exclusively with string content, they even provide useful extensions, such as a WriteLine(string) method that will terminate the string you've passed in with a line terminator character. Meanwhile, the ReadLine() method will return characters from your current index up to and including the next line terminator in the buffer. This isn't terribly useful with a serialized object, since you don't want to read a line of a JSON string. However, if you're working with a plain-text response, it can make reading and writing that response a breeze.

Seek vs Peek

Using the reader and writer types don't allow us to set the current index of the stream, for that we have to directly access the underlying stream using their BaseStream property.

var message = "stream test message";  
using Stream fs = new FileStream("strm.txt", FileMode.OpenOrCreate);  
using StreamWriter sw = new StreamWriter(fs);  
sw.BaseStream.Seek(10, SeekOrigin.Begin);  
sw.Write(message);

It's not uncommon to forward search through a string until arriving at a terminating character or flag value. Doing so with the StreamReader.Read() operation will result in moving the index past the terminating character and popping the terminating character off the array. If you want to simply read the last character before the terminating character, though, you have the Peek() operation.

Peek() will return the next character in the array without advancing the current index of the StreamReader.

using Stream fs = new FileStream(loc, FileMode.OpenOrCreate);  
using StreamReader sr = new StreamReader(fs);  
while (sr.Peek() >= 0) sr.Read();

The result of the above code is that the stream index will be exactly on the termination character. This also means that the stream is closed and write is unavailable.

The NetworkStream class

This class behaves exactly the same as the FileStream class. The only difference being that it's underlying type is an instance of the Socket class.
All the operations we did on the FileStream class work on this type as well and it can also be the BaseStream of an instance of the StreamReader/Writer classes we've seen.

This type will be important when you want to implement your own socket connections.

Most stream operations have concurrent variations as well. This greatly improves the performance of your application since some stream operations will be heavy and time consuming.

An example of how to work with networks using the stream types that we've learned so far with their concurrent counterparts:

public async Task<Result> PostRequestAsync() {  
    var result = new Result();  

    var request = WebRequest.Create("http://test-domain.com");  

    request.Method = "POST";

    Stream reqStream = request.GetRequestStream();  

    using StreamWriter sw = new StreamWriter(reqStream);

    await sw.WriteAsync("Our test data query");   

    var webResponse = await request.GetResponseAsync();  

    using StreamReader sr = new StreamReader(webResponse.GetResponseStream());

    result.RequestResult = await sr.ReadToEndAsync();

    return result; 
}

---

This concludes the basic stream foundations that we'll need when working with networks in .NET, There are other types that inherit Stream and are for other purposes, but I'm sure by now you've understood that all those types do, only convenients working with the basic underlying Stream type (just like we saw with StreamReader/Writer for strings).

There are other types that are specifically made for working with files and directories, but you can go and see their implementations, it all comes from streams.

I hope I could give some useful context to you!