keep asking why until the answer becomes physics
can a task that requires 6 billion cpu cycles every second run in real time on a single 5 ghz core?
no.
a 5 ghz core contains a maximum budget of:
`5 * 10^9 cycles per second`
the workload requires:
`6 * 10^9 cycles per second`
the utilization required is therefore:
`rho = demand / capacity = 6 / 5 = 1.2`
120%.
if the task is finite, it needs at least 1.2 seconds of cpu time.
if a new task arrives every second, the system accumulates 1 billion cycles of unfinished work every second. after ten seconds, the queue contains 10 billion cycles. latency does not stabilize at a bad value. it grows forever.
no runtime, compiler flag, allocator, thread pool, or programming language can make 6 billion clock cycles fit inside 5 billion clock cycles.
this is first-principles reasoning.
before asking how to optimize the code, write the budget equation. determine what the machine can physically provide, what the workload demands, and whether the two numbers can coexist.
but there is an important distinction.
saying that the current implementation consumes 6 billion cycles is not the same as proving that the task fundamentally requires 6 billion cycles.
the first number comes from a profiler.
the second requires a proof.
this is where most of the actual work begins.
why does the task consume 6 billion cycles?
because it processes 1 billion objects and spends 6 cycles on each.
why does it process 1 billion objects?
because it scans the complete dataset every second.
why does it scan the complete dataset?
because the result is recomputed from zero.
why is it recomputed from zero?
because the program does not maintain the result incrementally as new events arrive.
the physical limit was real. the workload description was not.
instead of making the loop 20% faster, the architecture can remove the loop.
if only one million objects changed since the previous result, an incremental system can process those one million changes instead of scanning one billion objects again. the problem moves from work proportional to the size of the complete dataset to work proportional to the number of changes.
the cpu did not become faster.
the task became smaller.
this is what optimization usually is at its deepest level. not asking the machine to violate its limits, but discovering that the work we asked it to perform was never necessary.
the unit also matters.
cycles are not instructions.
a modern cpu can execute several independent instructions during one clock cycle. it can issue operations to multiple execution units, overlap memory accesses, predict branches, and apply one vector instruction to multiple values at once.
if a workload requires 6 billion instructions per second, a 5 ghz core might be able to execute it with an average throughput above 1.2 instructions per cycle.
if it requires 6 billion measured core cycles per second, it cannot.
first-principles reasoning starts by refusing to compare quantities that only look equivalent.
instructions are not cycles.
cycles are not nanoseconds unless the frequency is known.
cpu time is not wall time.
throughput is not latency.
average capacity is not capacity under the worst case.
a single 5 ghz core also does not mean that every operation takes 0.2 nanoseconds.
`1 / 5,000,000,000 = 0.2 ns`
that is the duration of one clock period. an addition may have a latency of a few cycles. a dependent chain cannot use instruction-level parallelism. a branch misprediction can discard work already travelling through the pipeline. a cache miss can leave the core waiting for hundreds of cycles.
the clock gives the machine opportunities to make progress.
it does not guarantee useful work during every opportunity.
then memory enters the equation.
suppose a computation must inspect `d` bytes and the memory subsystem can deliver at most `b` bytes per second. even with free arithmetic, its execution time is bounded by:
`t >= d / b`
if the program must read 100 gb through a memory path delivering 50 gb/s, it cannot finish in less than two seconds.
rewriting the loop in rust, c, assembly, or handwritten machine code cannot move 100 gb through a 50 gb/s channel in one second.
but again, the first-principles question is not only whether the bandwidth limit exists.
it is whether those 100 gb must be read.
maybe the data is copied three times between layers.
maybe a large object representation contains eight bytes of information and sixty bytes of pointers, padding, metadata, and allocator overhead.
maybe the algorithm scans historical state when it only needs the latest state.
maybe the answer can be precomputed.
maybe the exact answer is unnecessary and an approximation would change the amount of information that must be processed.
the bandwidth limit is physical.
the number of bytes presented to it is often architectural.
the same reasoning applies to networks.
for a message travelling over distance `l` through a medium where the signal propagates at speed `v`:
`t >= l / v`
no software optimization can make information arrive before the physical signal.
if a protocol requires three sequential network round trips, its lower bound is at least three times the propagation delay before serialization, routing, congestion, kernel work, parsing, or application code are considered.
the code might take 50 microseconds.
the protocol might already require 100 milliseconds.
optimizing the code by 50% saves 25 microseconds from a system whose structure costs 100 milliseconds.
the only meaningful optimization is to change the number of round trips, move the computation closer to the data, cache the result, speculate before the request arrives, or change the consistency requirement.
once again, the machine is not the problem.
the dependency graph is.
physics appears inside the processor too.
increasing frequency means switching transistors more often. dynamic power is approximately:
`p = activity * capacitance * voltage^2 * frequency`
frequency increases power directly. reaching a higher frequency often also requires increasing voltage, which makes power rise faster because voltage is squared. that power becomes heat. the heat must leave the silicon fast enough for the chip to remain within its operating temperature.
signals also need time to cross wires and logic gates. every clock period must be long enough for the slowest required signal path to settle. at 5 ghz, the processor has 0.2 nanoseconds per period. the chip cannot be made arbitrarily larger, more complex, and faster while expecting every signal to arrive within an ever-smaller window.
this is why frequency did not simply continue from 5 ghz to 50 ghz to 500 ghz.
the limit is not a lack of ambition in processor design.
it is propagation, voltage, leakage, power density, and heat.
when one core is insufficient, parallelism increases the available budget.
with `p` cores running at frequency `f`, the theoretical capacity becomes:
`capacity = p * f`
but only for work that can be divided.
in practice:
`capacity = p * f * efficiency`
efficiency is below one because threads need to be scheduled, data needs to be partitioned, results need to be merged, and shared memory needs to remain coherent.
two logical threads on one physical core do not provide two complete 5 ghz cores either. they share execution units, caches, bandwidth, and other internal resources. simultaneous multithreading can fill unused parts of the pipeline. it cannot duplicate the silicon underneath it.
if the 6-billion-cycle workload is perfectly parallel, two cores can provide enough theoretical capacity.
if those 6 billion cycles form one dependent chain where every operation needs the result of the previous operation, adding cores does almost nothing.
the first-principles limit is then not total compute.
it is dependency depth.
this gives a useful hierarchy for every performance problem.
first, define the result that must exist.
then define the deadline.
then determine the minimum information that must be read.
then determine the minimum operations implied by that information.
then find the dependencies that force those operations to happen sequentially.
then compare the resulting lower bounds with compute throughput, memory bandwidth, network propagation, storage throughput, energy, and time.
everything above those bounds is implementation.
allocations can be removed.
copies can be removed.
branches can be reorganized.
data can be packed into cache lines.
scalar operations can become vector operations.
repeated computation can become cached state.
global synchronization can become local accumulation followed by one reduction.
exact results can sometimes become bounded approximations.
general-purpose processors can become specialized hardware.
but once the irreducible task requires more information, energy, sequential operations, or signal propagation than the physical system can provide before the deadline, the answer is no.
not difficult.
not expensive.
not waiting for a better framework.
impossible under the stated constraints.
the last three words matter.
when a result is impossible, change the constraints.
accept more latency.
use more cores.
move data closer.
reduce precision.
process fewer inputs.
maintain state incrementally.
change the algorithm.
change the hardware.
or stop requiring the result.
first principles is not the habit of declaring things impossible.
it is the habit of locating exactly where possibility ends.
keep asking why until the answer becomes an equation you cannot negotiate with.
everything above that equation is still a design choice.