DEV Community: Wesley

.NET: The ASCII Problem

Wesley — Sun, 31 Dec 2023 15:43:43 +0000

This might be seen as just unnecessary for optimizing binaries, but you'd have to at least ask yourself, why does .NET do what leads to these solutions?

For historical reasons, System.String uses the UCS-2 character encoding, that is, UTF-16 without surrogate pairs.
However, most strings in typical .NET applications consist solely of ASCII characters, leading to wasted space: half of the bytes in a string are likely to be null bytes!
Since strings are immutable, we can scan the character data when the string is constructed, then dynamically select an encoding, thereby saving 50% of string memory in most cases.

ASCII Mono | Mono

For some reason, this applies to all string literals except enum names and field names.

The fork linked in this page hasn't been active since 2016, and I know, before I even try, if I clone it and run make that there will be a million errors. So now, it's time to pull out my best gimmick in programming, hacks.

Tests

Here is a base executable with no code to execute:
3.0KB:

0000: 4D5A90000300000004000000FFFF0000 MZ..............
0010: B8000000000000004000000000000000 ........@.......
0020: 00000000000000000000000000000000 ................
0030: 00000000000000000000000080000000 ................
0040: 0E1FBA0E00B409CD21B8014CCD215468 ........!..L.!Th
0050: 69732070726F6772616D2063616E6E6F is program canno
0060: 742062652072756E20696E20444F5320 t be run in DOS
0070: 6D6F64652E0D0D0A2400000000000000 mode....$.......
0080: 504500004C0103000000000000000000 PE..L...........
0090: 00000000E00002010B01080000040000 ................
00a0: 0006000000000000BE22000000200000 ........."... ..
00b0: 00400000000040000020000000020000 .@....@.. ......
00c0: 04000000000000000400000000000000 ................
00d0: 00800000000200000000000003004085 ..............@.
00e0: 00001000001000000000100000100000 ................
00f0: 00000000100000000000000000000000 ................
0100: 702200004B00000000400000D0020000 p"..K....@......
0110: 00000000000000000000000000000000 ................
0120: 006000000C0000000000000000000000 .`..............
0130: 00000000000000000000000000000000 ................
0140: 00000000000000000000000000000000 ................
0150: 00000000000000000020000008000000 ......... ......
0160: 00000000000000000820000048000000 ......... ..H...
0170: 00000000000000002E74657874000000 .........text...
0180: C4020000002000000004000000020000 ..... ..........
0190: 00000000000000000000000020000060 ............ ..`
01a0: 2E72737263000000D002000000400000 .rsrc........@..
01b0: 00040000000600000000000000000000 ................
01c0: 00000000400000402E72656C6F630000 ....@..@.reloc..
01d0: 0C0000000060000000020000000A0000 .....`..........
01e0: 00000000000000000000000040000042 ............@..B
01f0: 00000000000000000000000000000000 ................
0200: A0220000000000004800000002000500 ."......H.......
0210: 5C2000000C0200000100000002000006 \ ..............
0220: 00000000000000000000000000000000 ................
0230: 00000000000000000000000000000000 ................
0240: 00000000000000000000000000000000 ................
0250: 1E02280100000A2A062A000042534A42 ..(....*.*..BSJB
0260: 01000100000000000C00000076322E30 ............v2.0
0270: 2E35303732370000000005006C000000 .50727......l...
0280: D0000000237E00003C01000084000000 ....#~..<.......
0290: 23537472696E677300000000C0010000 #Strings........
02a0: 0800000023555300C801000010000000 ....#US.........
02b0: 2347554944000000D801000034000000 #GUID.......4...
02c0: 23426C6F620000000000000002000010 #Blob...........
02d0: 471500000900000000FA013300160000 G..........3....
02e0: 01000000020000000200000002000000 ................
02f0: 01000000020000000100000001000000 ................
0300: 0100000000007B000100000000000600 ......{.........
0310: 17001E00060034005200000000000100 ......4.R.......
0320: 0000000001000100000010000A000000 ................
0330: 05000100010050200000000086182500 ......P ......%.
0340: 0100010058200000000096002B000500 ....X ......+...
0350: 01000000010012000900250001001100 ..........%.....
0360: 250001002E0013000B00048000000000 %...............
0370: 00000000000000000000000030000000 ............0...
0380: 0200000000000000000000002A007200 ............*.r.
0390: 0000000000000000003C4D6F64756C65 .........<Module
03a0: 3E0050726F6772616D0061726773004F >.Program.args.O
03b0: 626A6563740053797374656D002E6374 bject.System..ct
03c0: 6F72004D61696E007768790052756E74 or.Main.why.Runt
03d0: 696D65436F6D7061746962696C697479 imeCompatibility
03e0: 4174747269627574650053797374656D Attribute.System
03f0: 2E52756E74696D652E436F6D70696C65 .Runtime.Compile
0400: 725365727669636573006D73636F726C rServices.mscorl
0410: 6962007768792E657865000000032000 ib.why.exe.... .
0420: 000000001B8EF60B53E43C4E97C8F435 ........S.<N...5
0430: 7DB1B9050003200001050001011D0E1E }..... .........
0440: 01000100540216577261704E6F6E4578 ....T..WrapNonEx
0450: 63657074696F6E5468726F77730108B7 ceptionThrows...
0460: 7A5C561934E089000000000000000000 z\V.4...........
0470: 982200000000000000000000AE220000 ."..........."..
0480: 00200000000000000000000000000000 . ..............
0490: 0000000000000000A022000000000000 ........."......
04a0: 00005F436F724578654D61696E006D73 .._CorExeMain.ms
04b0: 636F7265652E646C6C0000000000FF25 coree.dll......%
04c0: 00204000000000000000000000000000 . @.............
04d0: 00000000000000000000000000000000 ................

At the bare minimum, executables have an extra 1.5KB of data for the .reloc and .rsrc section, which the latter has version info.
DEV is not helping me here with the little width I have to work with.

Plain String

Here's a plain call to WriteLine in Program.Main.

Console.WriteLine("Lorem ipsum dolor sit amet, consectetur...

Compile command: mcs -debug- -nostdlib- -o+ -sdk:2 test.cs
Binary view: (xxd -u -g 1 -s +0x200 -l 0x400 test.exe)
4.0KB:

03a0: 2F0084000000000000000000003C4D6F /............<Mo
03b0: 64756C653E0050726F6772616D006172 dule>.Program.ar
03c0: 677300436F6E736F6C65005379737465 gs.Console.Syste
03d0: 6D0057726974654C696E65004F626A65 m.WriteLine.Obje
03e0: 6374002E63746F72004D61696E007768 ct..ctor.Main.wh
03f0: 790052756E74696D65436F6D70617469 y.RuntimeCompati
0400: 62696C69747941747472696275746500 bilityAttribute.
0410: 53797374656D2E52756E74696D652E43 System.Runtime.C
0420: 6F6D70696C6572536572766963657300 ompilerServices.
0430: 6D73636F726C6962007768792E657865 mscorlib.why.exe
0440: 000000000084054C006F00720065006D .......L.o.r.e.m
0450: 00200069007000730075006D00200064 . .i.p.s.u.m. .d
0460: 006F006C006F00720020007300690074 .o.l.o.r. .s.i.t
0470: 00200061006D00650074002C00200063 . .a.m.e.t.,. .c
0480: 006F006E007300650063007400650074 .o.n.s.e.c.t.e.t
0490: 00750072002000610064006900700069 .u.r. .a.d.i.p.i
04a0: 007300630069006E006700200065006C .s.c.i.n.g. .e.l
04b0: 00690074002E0020004E0075006E0063 .i.t... .N.u.n.c
04c0: 00200065006700650074002000610075 . .e.g.e.t. .a.u
04d0: 00630074006F0072002000740065006C .c.t.o.r. .t.e.l
04e0: 006C00750073002C0020007500740020 .l.u.s.,. .u.t.
04f0: 0063006F006E00640069006D0065006E .c.o.n.d.i.m.e.n
0500: 00740075006D0020006500730074002E .t.u.m. .e.s.t..
0510: 0020004E0075006C006C006100200066 . .N.u.l.l.a. .f
0520: 00650072006D0065006E00740075006D .e.r.m.e.n.t.u.m
0530: 002C0020007400750072007000690073 .,. .t.u.r.p.i.s
0540: 002000730069007400200061006D0065 . .s.i.t. .a.m.e
0550: 0074002000680065006E006400720065 .t. .h.e.n.d.r.e
0560: 00720069007400200072007500740072 .r.i.t. .r.u.t.r
0570: 0075006D002C00200061006E00740065 .u.m.,. .a.n.t.e
0580: 00200064007500690020006C006F0062 . .d.u.i. .l.o.b
0590: 006F0072007400690073002000610075 .o.r.t.i.s. .a.u
05a0: 006700750065002C0020007300650064 .g.u.e.,. .s.e.d
05b0: 002000700075006C00760069006E0061 . .p.u.l.v.i.n.a
05c0: 007200200065007800200064006F006C .r. .e.x. .d.o.l
05d0: 006F00720020006E006F006E0020006C .o.r. .n.o.n. .l
05e0: 0061006300750073002E0020004E0061 .a.c.u.s... .N.a
05f0: 006D0020007300650064002000650073 .m. .s.e.d. .e.s

This string in binary takes the size of an EXE section. (0x400 bytes / 1KB)
This is doomsday for large console apps.

String Resource

Loading a string from a resource:

Console.WriteLine(Resources.ResourceManager.GetString("test"));

Resource file:

<?xml version="1.0" encoding="utf-8"?>
<root>
    ...
    <data name="test" xml:space="preserve">
        <value>Lorem ipsum dolor sit amet, consectetur...</value>
    </data>
</root>

Command: resgen /useSourcePath /compile main.resx
Added to MCS args: -resource:main.resources
4.5KB:

0200: D0280000000000004800000002000500 .(......H.......
0210: 80230000200500000100000004000006 .#.. ...........
0220: AC200000D20200000000000000000000 . ..............
0230: 00000000000000000000000000000000 ................
0240: 00000000000000000000000000000000 ................
0250: 1E02280900000A2AD27E010000041428 ..(....*.~.....(
0260: 0500000A391E0000007201000070D002 ....9....r...p..
0270: 000002280600000A6F0700000A730800 ...(....o....s..
0280: 000A80010000047E010000042A1E0228 .......~....*..(
0290: 0900000A2A5628020000067209000070 ....*V(....r...p
02a0: 6F0A00000A280B00000A2A00CE020000 o....(....*.....
02b0: CECAEFBE01000000910000006C537973 ............lSys
02c0: 74656D2E5265736F75726365732E5265 tem.Resources.Re
02d0: 736F757263655265616465722C206D73 sourceReader, ms
02e0: 636F726C69622C2056657273696F6E3D corlib, Version=
02f0: 342E302E302E302C2043756C74757265 4.0.0.0, Culture
0300: 3D6E65757472616C2C205075626C6963 =neutral, Public
0310: 4B6579546F6B656E3D62373761356335 KeyToken=b77a5c5
0320: 3631393334653038392353797374656D 61934e089#System
0330: 2E5265736F75726365732E52756E7469 .Resources.Runti
0340: 6D655265736F75726365536574020000 meResourceSet...
0350: 00010000000000000050414450414450 .........PADPADP
0360: 33AF737C00000000C900000008740065 3.s|.........t.e
0370: 0073007400000000000182044C6F7265 .s.t........Lore
0380: 6D20697073756D20646F6C6F72207369 m ipsum dolor si
0390: 7420616D65742C20636F6E7365637465 t amet, consecte
03a0: 7475722061646970697363696E672065 tur adipiscing e
03b0: 6C69742E204E756E6320656765742061 lit. Nunc eget a
03c0: 7563746F722074656C6C75732C207574 uctor tellus, ut
03d0: 20636F6E64696D656E74756D20657374  condimentum est
07a0: 0000000000000000B700230100000000 ..........#.....
07b0: 0000000001000000E401000000000000 ................
07c0: 003C4D6F64756C653E00520050726F67 .<Module>.R.Prog
07d0: 72616D00724D005265736F757263654D ram.rM.ResourceM
07e0: 616E616765720053797374656D2E5265 anager.System.Re
07f0: 736F75726365730047656E6572617465 sources.Generate
0800: 64436F64654174747269627574650053 dCodeAttribute.S
0810: 797374656D2E436F6465446F6D2E436F ystem.CodeDom.Co
0820: 6D70696C6572002E63746F7200446562 mpiler..ctor.Deb
0830: 75676765724E6F6E55736572436F6465 uggerNonUserCode
0840: 4174747269627574650053797374656D Attribute.System
0850: 2E446961676E6F737469637300436F6D .Diagnostics.Com
0860: 70696C657247656E6572617465644174 pilerGeneratedAt
0870: 747269627574650053797374656D2E52 tribute.System.R
0880: 756E74696D652E436F6D70696C657253 untime.CompilerS
0890: 6572766963657300456469746F724272 ervices.EditorBr
08a0: 6F777361626C65417474726962757465 owsableAttribute
08b0: 0053797374656D2E436F6D706F6E656E .System.Componen
08c0: 744D6F64656C00456469746F7242726F tModel.EditorBro
08d0: 777361626C655374617465004F626A65 wsableState.Obje
08e0: 63740053797374656D00526566657265 ct.System.Refere
08f0: 6E6365457175616C7300547970650047 nceEquals.Type.G
0900: 65745479706546726F6D48616E646C65 etTypeFromHandle
0910: 0052756E74696D655479706548616E64 .RuntimeTypeHand
0920: 6C65006765745F417373656D626C7900 le.get_Assembly.
0930: 417373656D626C790053797374656D2E Assembly.System.
0940: 5265666C656374696F6E006172677300 Reflection.args.
0950: 476574537472696E6700436F6E736F6C GetString.Consol
0960: 650057726974654C696E65006765745F e.WriteLine.get_
0970: 4D004D61696E004D007768790052756E M.Main.M.why.Run
0980: 74696D65436F6D7061746962696C6974 timeCompatibilit
0990: 79417474726962757465006D73636F72 yAttribute.mscor
09a0: 6C6962007768792E7265736F75726365 lib.why.resource
09b0: 73007768792E65786500000000077700 s.why.exe.....w.
09c0: 68007900000974006500730074000000 h.y...t.e.s.t...
09d0: 7855E192F3A8DF4ABD3B8148F7C608B2 xU.....J.;.H....
09e0: 0003061205052002010E0E4101003353 ...... ....A..3S
09f0: 797374656D2E5265736F75726365732E ystem.Resources.
0a00: 546F6F6C732E5374726F6E676C795479 Tools.StronglyTy
0a10: 7065645265736F757263654275696C64 pedResourceBuild
0a20: 65720831372E302E302E300000032000 er.17.0.0.0... .

This adds nearly as much bloat with the bottom data.

String as Byte[]

(Real world example)
(sigh):

Console.WriteLine(Encoding.ASCII.GetString(
    new byte[] {(byte)'L',(byte)'o',(byte)'r',(byte)'e',(byte)'m',(byte)' ',(byte)'i'...}
));

5.0KB (...):

0100: F02300004B00000000600000D0020000 .#..K....`......
0110: 00000000000000000000000000000000 ................
0120: 008000000C0000000000000000000000 ................
0130: 00000000000000000000000000000000 ................
0140: 00000000000000000000000000000000 ................
0150: 00000000000000000020000008000000 ......... ......
0160: 00000000000000000820000048000000 ......... ..H...
0170: 00000000000000002E74657874000000 .........text...
0180: 44040000002000000006000000040000 D.... ..........
0190: 00000000000000000000000020000060 ............ ..`
01a0: 2E736461746100000402000000400000 .sdata.......@..
01b0: 00040000000A00000000000000000000 ................
01c0: 00000000400000C02E72737263000000 ....@....rsrc...
01d0: D00200000060000000040000000E0000 .....`..........
01e0: 00000000000000000000000040000040 ............@..@
01f0: 2E72656C6F6300000C00000000800000 .reloc..........
0200: 00020000001200000000000000000000 ................
0630: 03000000003C4D6F64756C653E005072 .....<Module>.Pr
0640: 6F6772616D0061726773004279746500 ogram.args.Byte.
0650: 53797374656D003C5072697661746549 System.<PrivateI
0660: 6D706C656D656E746174696F6E446574 mplementationDet
0670: 61696C733E0024417272617954797065 ails>.$ArrayType
0680: 3D35313600246669656C642D31323333 =516.$field-1233
0690: 38454237413930433932343238313343 8EB7A90C9242813C
06a0: 41463436453833353941303343304242 AF46E8359A03C0BB
06b0: 444133440052756E74696D6548656C70 DA3D.RuntimeHelp
06c0: 6572730053797374656D2E52756E7469 ers.System.Runti
06d0: 6D652E436F6D70696C65725365727669 me.CompilerServi
06e0: 63657300496E697469616C697A654172 ces.InitializeAr
06f0: 7261790041727261790052756E74696D ray.Array.Runtim
0700: 654669656C6448616E646C6500436F6E eFieldHandle.Con
0710: 736F6C650057726974654C696E65004F sole.WriteLine.O
0a00: 4C6F72656D20697073756D20646F6C6F Lorem ipsum dolo
0a10: 722073697420616D65742C20636F6E73 r sit amet, cons
0a20: 65637465747572206164697069736369 ectetur adipisci
0a30: 6E6720656C69742E204E756E63206567 ng elit. Nunc eg
0a40: 657420617563746F722074656C6C7573 et auctor tellus

This adds an extra section named ".sdata" where the byte array is stored, and a stupidly long field name for the array.

Enum

Probably the stupidest way to work around this, but could only work in certain circumstances like storing/using config keys.
(Real world example
Rough version that doesn't use a proper array or support punctuation:

public enum _ {
    Lorem,
    ipsum,
    dolor,
    sit,
    amet,
    consectetur,
    ...
    (dupes have underscore appended :(   )
}
string test = "";
for (int i = 0; i < (int)(_._); i++)
    test += ((_)i).ToString() + ' ';
Console.WriteLine(test);

5.0KB:

03a0: 09000100010002010000120000001500 ................
03b0: 0100030006061400010056801C000400 ..........V.....
03c0: 56802200040056802800040056802E00 V."...V.(...V...
03d0: 04005680320004005680370004005680 ..V.2...V.7...V.
03e0: 4300040056804E000400568053000400 C...V.N...V.S...
03f0: 56805800040056805D00040056806400 V.X...V.]...V.d.
0400: 040056806B00040056806E0004005680 ..V.k...V.n...V.
0410: 7A00040056807E000400568084000400 z...V.~...V.....
0420: 56808E00040056809500040056809A00 V.....V.....V...
0430: 04005680A00004005680AA0004005680 ..V.....V.....V.
0440: B10004005680B60004005680BA000400 ....V.....V.....
0450: 5680C30004005680C90004005680CD00 V.....V.....V...
0770: 5201080014015701080018015C010800 R.....W.....\...
0780: 1C016101080020016601080024016B01 ..a... .f...$.k.
0790: 08002801700108002C01750108003001 ..(.p...,.u...0.
07a0: 7A01080034017F010800380184010800 z...4.....8.....
07b0: 3C018901080040018E01080044019301 <.....@.....D...
07c0: 2E003300BC01B5010480000000000000 ..3.............
07d0: 00000000000000000000780200000200 ..........x.....
07e0: 00000000000000000000DB01BA020000 ................
07f0: 0000030002000000003C4D6F64756C65 .........<Module
0800: 3E0050726F6772616D005F0076616C75 >.Program._.valu
0810: 655F5F004C6F72656D00697073756D00 e__.Lorem.ipsum.
0820: 646F6C6F720073697400616D65740063 dolor.sit.amet.c
0830: 6F6E7365637465747572006164697069 onsectetur.adipi
0840: 7363696E6700656C6974004E756E6300 scing.elit.Nunc.
0850: 6567657400617563746F720074656C6C eget.auctor.tell
0860: 757300757400636F6E64696D656E7475 us.ut.condimentu
0870: 6D00657374004E756C6C61006665726D m.est.Nulla.ferm
0880: 656E74756D0074757270697300736974 entum.turpis.sit

This takes up 0x440 bytes.

Raw ASCII string squished into UTF-16 LE C# String

and then converted to a byte array with Unicode GetBytes,
and then converted to a string with ASCII GetString.

Console.WriteLine(Encoding.ASCII.GetString(Encoding.Unicode.GetBytes(
    "潌敲⁭灩畳⁭潤潬⁲楳⁴浡瑥‬潣獮捥整畴⁲摡灩獩楣杮攠..."
)));

3.5KB(!!):

03a0: 750017002100750017002E003B002100 u...!.u.....;.!.
03b0: 04800000000000000000000000000000 ................
03c0: 00008000000002000000000000000000 ................
03d0: 00004000C200000000000000003C4D6F ..@..........<Mo
03e0: 64756C653E0050726F6772616D006172 dule>.Program.ar
03f0: 677300456E636F64696E670053797374 gs.Encoding.Syst
0400: 656D2E54657874006765745F41534349 em.Text.get_ASCI
0410: 49006765745F556E69636F6465004765 I.get_Unicode.Ge
0420: 74427974657300476574537472696E67 tBytes.GetString
0430: 00436F6E736F6C650053797374656D00 .Console.System.
0440: 57726974654C696E65004F626A656374 WriteLine.Object
0450: 002E63746F72004D61696E0077687900 ..ctor.Main.why.
0460: 52756E74696D65436F6D706174696269 RuntimeCompatibi
0470: 6C697479417474726962757465005379 lityAttribute.Sy
0480: 7374656D2E52756E74696D652E436F6D stem.Runtime.Com
0490: 70696C65725365727669636573006D73 pilerServices.ms
04a0: 636F726C6962007768792E6578650000 corlib.why.exe..
04b0: 0082034C6F72656D20697073756D2064 ...Lorem ipsum d
04c0: 6F6C6F722073697420616D65742C2063 olor sit amet, c
04d0: 6F6E7365637465747572206164697069 onsectetur adipi
04e0: 7363696E6720656C69742E204E756E63 scing elit. Nunc
04f0: 206567657420617563746F722074656C  eget auctor tel
0500: 6C75732C20757420636F6E64696D656E lus, ut condimen
0510: 74756D206573742E204E756C6C612066 tum est. Nulla f
0520: 65726D656E74756D2C20747572706973 ermentum, turpis
0690: 612C20657420626962656E64756D2066 a, et bibendum f
06a0: 656C69732074656D7075732061636375 elis tempus accu
06b0: 6D73616E2E0100003EF604CCBDC0D640 msan....>......@
06c0: B4394801D5D5B2C90004000012050520 .9H............
06d0: 011D050E0520010E1D05040001010E03 ..... ..........
06e0: 200001050001011D0E1E010001005402  .............T.
06f0: 16577261704E6F6E457863657074696F .WrapNonExceptio
0700: 6E5468726F77730108B77A5C561934E0 nThrows...z\V.4.
0710: 89000000000000000000000000000000 ................
0720: 4825000000000000000000005E250000 H%..........^%..
0730: 00200000000000000000000000000000 . ..............
0740: 00000000000000005025000000000000 ........P%......
0750: 00005F436F724578654D61696E006D73 .._CorExeMain.ms
0760: 636F7265652E646C6C0000000000FF25 coree.dll......%
0770: 00204000000000000000000000000000 . @.............
0780: 00000000000000000000000000000000 ................

The string takes 0x200 bytes / 0.5 KB.

"fixing" physical "free space" on Windows

Wesley — Mon, 30 Oct 2023 19:34:02 +0000

I don't know if this is just my computer or my hard drive or something stupid, but here I am to write about it:

Notice anything unnatural about large files you have? For example, this (NTFS compressed) video that's 500 MB.

On the disk, it should physically take up 500 MB, right?

APPARENTLY NOT.

(it takes up 600 MB at the start of copying, and then spends the rest of the 400MB while the progress bar loads... !?!?!?!?!?!?!)

Maybe because I have it compressed. That makes no sense, but I wonder what happens if that's turned off.

Huh? I gained space?? What if I recompress it?

?????
(this apparently doesn't happen on my SSD)

Copying the uncompressed file in a folder that does not have compression turned on makes it not take up more space than it really does. !?!?!?!?!?!?!?!?!?!?!?!

To investigate further, I made a little shell script (with Cygwin) (a little broken, though) that takes files/folders as arguments, and for each individual file, read the total amount of free space of the disk the file is on before and after running Windows' COMPACT command, and if space was taken up after uncompressing it or decompressing it (depending on the attribute it started off with), revert the change.
However, the above example proved this to be made slightly useless because only a select few files actually took more space due to being compressed, which was the motive for making this script.
In most of my use cases involving hundreds of files, it gradually is able to free little amounts of space. I've tested this on other big video files and a few gigabytes were freed up.

Download script
(requires Cygwin, should work on a POSIX shell like dash)

Download Real Time Disk Size Monitor

Compressing a folder of music:

I've held off publishing this for about a week now probably because I don't know a conclusion for this article, but I can just say ALL OF THIS MAKES NO SENSE!!!!!!!!

oh-ex-oh: automated 0x0.st uploading and file management

Wesley — Sun, 03 Sep 2023 13:41:20 +0000

https://github.com/donnaken15/oh-ex-oh

As if just curl -F'file=@mystupidfile.png' https://0x0.st wasn't simple enough.

This is a basic singular shell script that can take one file as an argument and upload it. Woweee. But it can also save tokens given from the file hoster so you can set an expiry date or manually delete it at a later time after uploading.

This was kind of made to test my hand at full on shell scripting as well because half of the work in this is a custom config reader/writer, option parsing. I haven't tested this other thing yet, but I made it in a way for the host for uploads to be changeable so another 0x0 instance can be used.

Soon, I'll want to get an interface working for this but still in the terminal, and accepting piped input.

Made to be compatible with zsh, so not technically bash, trying for this to be as POSIX as I can without weird hacks.
Tested on Cygwin.

Timelapse of code

File uploaded in above example: https://0x0.st/HpwD.gif
(I'm able to reupload the file under the same link if the file stays the same)

CALU: a basic char based arithmetic calculator (in progress)

Wesley — Sat, 01 May 2021 16:20:42 +0000

A little test program I'm writing that's not dependent on actual integers but rather using string based numbers, breaking the 4.29b/18.4t limit
It's being built for Windows and Linux, and is completely C standard library compliant
So far, I just got fully working addition to work and it will only do math with positive numbers
The way digits work is dynamic but have a static limit, so for example, 100+100000 shifts the digits for 100 to the right
These number strings can also have padded zeroes
Currently, I have this program set to support 256 digits

code:
https://github.com/donnaken15/CALU/blob/main/CALU.C
man:
https://github.com/donnaken15/CALU/blob/main/CALU.1

Cloneoa: the Klonoa custom level engine (indev)

Wesley — Sat, 13 Feb 2021 19:48:58 +0000

Here's something I started but immediately abandoned two years ago, but picked up again recently, solely for the fact of this game being very underrated for its potential that has yet to be unlocked, especially for its Mario Maker level potential

There's also already a level editor though called klo-gba.js by macabeus, but overall, Klonoa hacking and all that is still young and unfounded for whatever reason

What's currently available is level loading, platforming, some items, a music and level theme, and two enemies

The rest of what the original Empire of Dreams had will be added underway

I've also documented most of how level data will work, along with working on a graphical interface program to edit levels easier

Soon, I will also be opening a forum dedicated to this project to allow sharing creations and show development of the engine

Follow the development here
https://github.com/donnaken15/Cloneoa
or here
https://donnaken15.itch.io/klo

Community Discord:
https://discord.gg/fCzKFuEg3K

Autocommitting

Wesley — Sun, 29 Nov 2020 23:58:14 +0000

To resolve my issue for posting more progress in such major files infrequently, so my work can be published more regularly, without thinking to open GitHub Desktop, I got off the Batch mindset, shoved my brain and face into an asylum of the MSYS2 terminal testing all types of things, to make my first Shell script, dedicated to... you already read the title.

Anyway, with this script, it will help you to commit more often (if you don't already), by assigning it to a task scheduler or cron job to be run on a consistent schedule. Its usage is simply autocmt.sh ~/Documents/repos/123/.
This script will also name your commits by summarizing your additional changes in a compact manner.

For example, you can set this to run every day, and on other days, push your changes, which can be done by appending your command with -p.

Project page:
https://github.com/donnaken15/Autocommit

Direct script file

Warning, this was only tested with MINGW and WSL, so I don't know if it will work purely on Linux. Titles may not look the best or probably won't work AT ALL, either, but I tried, with multiple substring problems.
In total, I think this project took a third of a day, within two days.

Here's my repo I tested this script against while putting the finishing touches, it went well as you will see:
https://github.com/donnaken15/AutocmtTest/commits/master

Minimalist audio videos for songs

Wesley — Sat, 15 Aug 2020 01:35:28 +0000

So a while ago I found out about making one frame videos by simply throwing in an image, audio, and setting the framerate closest to the length of audio, in which I showed an example with Pinback's X.I.Y. for two minutes: https://www.youtube.com/watch?v=RIhPf-LqY6M
Which the command is basically:
ffmpeg -loop 1 -r 0.002777 -i cover.jpg -i audio.mp3 -ab 256k -shortest -t 360 video.avi
And of course is more efficient than 25fps or whatever fps for something that's just audio, and coincidentally as I was doing it with another album, YouTube Topic videos started just showing the album cover in a square rather than also including text, and having it at 25fps 🤔🤔🤔🤔🤔
I know someone will probably bring up the point of it not mattering how low the FPS is because of how the same frame is displayed for 4 minutes, but it actually does affect the file size, as I've tested a few times
Now at first, I was using rounded up numbers from this little online tool: http://www.zapstudio.net/framecalc/ and repeatedly changing numbers to see it get to the closest length of the audio until I found out the equation is 1 divided by the length of the song
I also happened to make a javascript function at the beginning of this year that takes in a string of HH:MM:SS and makes it a float (link)
With this I was able to mass render videos faster of audio from the album I mentioned, and not overestimate or underestimate 1 frame lasting 2-3 minutes
~~I might not be running a channel to upload a hardcore artist's entire catalog~~
So the process was simply get exact length of a song, 1/timeToFlt("00:04:40.250") in a browser console, and doing the same ffmpeg command as above but with the new framerate
And in addition, there was less to blank endings of the video after the song had already ended (and I noticed because the songs I was converting hit an instant stop instead of fading out normally)
Though it doesn't help YouTube will kind of force a minimal framerate anyway at like 5 or 8fps
In some cases (sometimes on mobile), it's so minimal that YouTube won't even show the quality setting, and I saw at some point seeking thumbnails weren't showing for a video or two

(Feb'19) Get Unicode Character Name in C# (DllImport) (charmap mechanism)

Wesley — Mon, 27 Jul 2020 07:38:18 +0000

Found this using IDA 6.8 Pro
Charmap loads a DLL named GetUName.dll which has a selfnamed function and pulls the name of a unicode character using an int and loads the name into a string
In C++, it is used like GetUName(int, LPWSTR *)
In C#, it can be imported this way:

[DllImport("GetUName.dll", CallingConvention = CallingConvention.StdCall)]
public static extern int GetUName(int x, [MarshalAs(UnmanagedType.LPWStr)]string lpBuffer);

Then, create a string buffer to load name in (255 in case of large names, can be changed)
public static string charactername = new string('*', 255);
And use in the program like so:

GetUName('A', charactername);
// asteriks are still left in (* because names don't include
// any), simply cut the length so they aren't included
charactername = charactername.Substring(0, charactername.IndexOf('*') - 1);
Console.Write(charactername);

Included this in a program I wrote that inserts a character in the previously active window: https://github.com/donnaken15/quickcharmap/
Also shared this on the pinvoke wiki:
https://www.pinvoke.net/default.aspx/getuname.GetUName

(6/21/20) Bit . Trip Editor: External code update

Wesley — Sat, 25 Jul 2020 22:19:00 +0000

Project @ https://github.com/donnaken15/Bit.Trip_Editor

Now opening all the stuff in my project file out of the box, and maybe sprites too

also since GitHub can detect GML now

some scripts may not be editable or exported
including beats which ill resolve soon

some other changes:
compressed game icons to minimal size and colors programmatically
real time code changes can be applied but it will be slower (turn on in settings with RealtimeSrc=1)
some others i forgot BECAUSE FIREFOX AND MY COMPUTER SUCKS

todos:
optimize beat beat functions, i might've went full yandev not inheriting code :-|
MAKE THE FREAKING VIOLET BEAT, THANKS GAMEMAKER
use xin instead of implemented functions, also maybe include wiimote support (GiiMote extension is still broken lol)
allow relative and absolute paddle control for mouse and joysticks (part of it done now)
figure out alarm 0-11 and some other events with object_event_add (OR JUST PUT IT IN STEP YOU STUPID KI-)
maybe get around optimizing font sprites and proportions and spacing because wasting 4 resized sprites of the same thing
export code and add by files for some other things i haven't already, will be a wreck to go through
maybe protect variables or objects from being messed up when having id names that people might accidentally overwrite
remove blank sprites for collision if it really isnt needed and since already drawing plain color rectangles
remove sprites of designer that can be replaced with line and rectangle drawing code
try using exitprocess or something to return certain game statuses in error codes
maybe dont load what doesnt need to (like not keeping in designer scripts or other things while playing a level)
the js port because ecma is trash and close to gamemakers free breakable syntax and because its web trash
edit menu, playtest, timeline/editor buttons, and object adding in designer

also added installer with automated registry stuff and adjusting to install location: github.com/donnaken15/Bit.Trip_Editor/releases/tag/installer

Megatouch 11UP in Javascript

Wesley — Tue, 14 Jul 2020 01:00:40 +0000

Porting my remake of Merit's tabletop 11UP for Windows in C++ to Javascript for browsers and phones.
Was kind of easy to port some basic things considering the syntax allowed
Obviously not complete just yet
Can be played and is updated live at https://donnaken15.github.io/11UP along with the Windows game being downloadable there, too
Feel like I don't play this too much because it's really fun sometimes

lookcool : conabuse (RNG based fake logger)

Wesley — Sun, 12 Jul 2020 19:49:08 +0000

One of a few programs I'm working on that create fun hacky visuals
Here's an example of the tool put to some cool music: https://www.youtube.com/watch?v=K5_pqIHrFJM
Project: https://github.com/donnaken15/lookcool/tree/master/conabuse