Energy Estimation query
Hi Ecologits Team, It's great to see you guys driving sustainability awareness in this fast moving space.
I have a generic query of sorts - Mainly to validate that my usage is valid/appropriate.
Using the calculator in expert model, and selecting the model, and then say for example entering an arbitrary completion token amount of 12,622,633, which overrides the example prompt input var/value - Can I assume this is an appropriate use of the calculator i.e.
That the example prompt is overriden by the completion token number
That the calculator accomodates large completion token values i.e. in the millions
I get an answer of c.843KWh which is large - not that I have any better calculation/ideas as to what it should be.
Thanks in advance.
Leo
Hey @leolion00 , thanks for the nice feedback :)
Yes that's correct, by doing so you asked for the impacts of one request outputing 12 622 633 tokens (which is +- 50M characters) using ChatGPT (if you didn't change the model).
Note however that such output is not realistic as ChatGPT has a limit of output tokens, but I guess that's not the point here!
Adrien
Hey
@adrienbanse
, thank you for the reply.
That completion token number is a sum total of completion tokens over the course of a month/many-many inferences/requests.
Does that numerically impact the total energy sum, as opposed to, if I were to plug-in many differing smaller inferences, with the sum total then being 12 622 633 for all of those inferences?
Many thanks,
Leo
Hey
@leolion00
,
Unfortunately yes, it does make a difference (as you can see in the calculator, if you take 2 000 output tokens and then 2 000 000, you won't have exactly 1 000x the energy).
Thanks @adrienbase- This is very helpful and thank you for your help thus far.
With the achknowledgement that just entering one very large monthly_completion_token_number (12_689_622) skews the numbers.
I have created a rather crude python routine, which uses ecologits to mimic the calculator to a degree.
Essentially I take the completion tokens and divide them by the requests to get an average number of tokens per request (805) and then simulate running that config/prompt through the routine 15762 times, to get a sum total of energy for all of those 805 token requests.
Does that make sense?
If I enter the same config into the calculator (openai, gpt 40, 805 tokens, united kingdon) and then multiply it by 15762, I get pretty close (1,109.64 kWh) to the number below (1,148.45 KWh) which is c.3% difference.
Iteration Sample Output
Request 3284: Energy = 0.0630 to 0.0729 kWh | Emissions = 0.0381 to 0.0440 kg CO₂e
Request 3285: Energy = 0.0630 to 0.0729 kWh | Emissions = 0.0381 to 0.0440 kg CO₂e
Request 3286: Energy = 0.0630 to 0.0729 kWh | Emissions = 0.0381 to 0.0440 kg CO₂e
Output
--- Monthly Summary ---
Total completion tokens: 12,689,622
Total requests: 15762
Expert model: ON
Total Energy: 993.56 to 1148.45 kWh
Total Emissions: 600.85 to 694.11 kg CO₂e
Average per request: 0.0630 to 0.0729 kWh | 0.0381 to 0.0440 kg CO₂e
Yes I would say that this is a good use of the tool! Let's just say that since there's already some approximation in the methodology we should try to stick to the most realistic use..
Just a word about the zone. I see you chose UK, but making your requests from the UK doesn't mean that the computations are ran in UK. In fact, I think that all of OpenAI's datacenters are in the US. So, unless (for some reason) you know that you use UK datacenters, I would advise to switch to the US or World's mix.
Brilliant - Thank you @adrienbanse ... I had incorrectly assumed UK/London based DCs.
Thanks a million for your help.
Leo