REMOID ACTIVATION FUNCTION Version: v0.3 Author: William Loffler Scratch Username: Willalex555 Created: July 2026 ------------------------------------------------------------ ABSTRACT REMOID is an experimental neural network activation function designed while developing MiniGPT, a transformer implemented entirely in Scratch. The goal of REMOID is to preserve gradients for negative values while preventing positive activations from growing completely unchecked. Unlike Sigmoid or Tanh, REMOID does not compress values into a fixed range. Instead, larger positive values are reduced gradually while still allowing the output to increase. REMOID is intended for experimentation and educational AI research. ------------------------------------------------------------ MOTIVATION Many modern neural networks use activation functions such as ReLU, Leaky ReLU, GELU, SiLU, and Sigmoid. Each has advantages and disadvantages. ReLU • Extremely simple • Fast • Unlimited positive growth Leaky ReLU • Prevents dead neurons • Still allows unlimited positive growth Sigmoid • Smooth • Compresses outputs into a fixed range • Can suffer from vanishing gradients REMOID was created to explore an alternative approach. Goals: • Preserve information from negative values. • Compress positive values without limiting them to a fixed maximum. • Remain computationally inexpensive. • Be practical for Scratch-based neural networks and transformers. ------------------------------------------------------------ FUNCTION If x is below 0 output = input * 0.01 Otherwise output = input - (input / (digit count + input * 0.9)) digit count = number of digits in floor(input) The decimal part is ignored. ------------------------------------------------------------ SCRATCH IMPLEMENTATION Negative Branch output = input * 0.01 Positive Branch output = input - ( input / ( length of (floor of (input)) + (input * 0.9) ) ) ------------------------------------------------------------ DESIGN GOALS REMOID was designed to: • Preserve negative gradients. • Compress larger positive activations. • Continue increasing rather than saturating. • Remain efficient enough for Scratch. • Be suitable for educational transformer projects. ------------------------------------------------------------ EXAMPLE OUTPUTS Input Output -10 -0.10 -2 -0.02 0 0 2 1.00 2.1 1.08 5 3.57 10 8.57 100 98.11 1000 998.02 ------------------------------------------------------------ ADVANTAGES • Simple implementation. • Does not completely flatten positive activations. • Compresses larger values more than smaller values. • Uses only basic arithmetic and digit counting. • Easily implemented inside Scratch. • Does not require exponentials or expensive operations. ------------------------------------------------------------ CURRENT STATUS REMOID is currently experimental. Future work includes: • Benchmarking against ReLU. • Benchmarking against Leaky ReLU. • Benchmarking against GELU. • Benchmarking against SiLU. • Measuring MiniGPT training performance. • Refining the compression function if needed. ------------------------------------------------------------ VERSION HISTORY v0.1 • Initial REMOID prototype. v0.2 • Switched to using floor() to ignore decimal places. • Simplified Scratch implementation. • Improved handling of decimal inputs. • Reduced overall block count. ------------------------------------------------------------ LICENSE REMOID is free for anyone to use in research, educational projects, commercial software, and open-source projects. Credit is appreciated. Suggested attribution: "REMOID Activation Function created by William Loffler (Scratch: Willalex555)." ------------------------------------------------------------ ACKNOWLEDGEMENTS REMOID was independently designed while developing MiniGPT, a transformer built entirely in Scratch as an educational AI project. The purpose of REMOID is to explore new activation function behavior in lightweight neural networks and encourage experimentation within the Scratch community and beyond.
