How to determine & monitor changes in frequency of an irregularly occuring event over time?

[TheGreenLama]

My question sounds fairly basic, but as statistics is not my strong suit I'm asking it here.

I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.

Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?

 

[GregLocock]

Funnily enough Hurricanes was my first thought. EMA is ideal for this. I'd use say 120 months as a first guess. That is,

f(t)=(119*f(t-1)+s(t))/120

where f is the ema and s is your time history of events. The advantages of ema over simple moving average f(t)=ave(s(t),s(t-1)...s(t-119)) is that it has calculable effects on the frequency response, and responds immediately to a spike at t even if there is a spike at t-120, and generally looks more like a real system. SMA is fine where there are a lot of events in the sample time, but in signal analysis it would be unusual to use a uniform weighted SMA - I'd only deliberately use it in strange circumstances.



Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[TheGreenLama]

I'm still a little fuzzy on the terms. You've introduced a set time, 120, when talking about EMA. I thought EMA was an infinite series going back to include all terms.

Isn't the unknown term introduced alpha (0 < alpha < 1)? With the corresponding equation something like Link:

where p would be the "time gaps" going back in the series. If so, how do you go about determining alpha?

To me this equation looks odd on the surface, because you're not dividing the sum of all of the "time gaps" by the total number of events.

 

[GregLocock]

f(t-1) includes some memory of all the preceeding values . alpha is 1/120 in the case I chose.

Just try it out, it works.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[TheGreenLama]

OK, so if I'm understanding your equation correctly, this tranlates to:
EMA(t) = EMA(t-1)*(119/120) + s(t)/120
where s(t) = latest time gap in question (what I've noted above as "p").

From that link I mentioned above, an alternate form of the above equation (using similar terms) is:
EMA(t) = EMA(t-1) + [s(t) - EMA(t-1)]/120

So, if you'll indulge me a little longer, a couple more questions:
1) These two equations appear to be slightly different. Any chance you have a reference for yours?
2) Hmm. Seems like I'm still a little vague on what equation to use for EMA(t-1). Is it included within the article I've linked to above?
3) What variables go into choosing an alpha? And, can I use this alpha from the very beginning of the event series, or do I need to wait until a certain time elapses before I can start calculating EMAs?

Thanks

 

[GregLocock]

last bit first, yes, you have to do something nasty until you have 1/alpha samples.

I'd have thought it was obvious that the two equations are identical, the one is just the other expanded.

No, no refs. Oh, well wiki has it all

https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average,

including the expansion.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[IRstuff]

What exactly is the statistical behavior of these events? Are they supposed to be trendable?

There are other options as well, depending on what is the nature of the problem. One possibility is a maximum likelihood estimator (MLE) like a full-blown Kalman filter, although your problem might only require an alpha-beta filter.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[TheGreenLama]

Chuckle, chuckle. Greg, you are correct. The equations are the same. My only defense is that it was late at night when I posted.

As far as the statistical behavior of the events, I'm not exactly sure. I'm guessing they are random, or should be random. I want to know if they are in fact random, or are the frequencies changing over time, and by how much. What does "supposed to be trendable" mean? Are some events non-trendable? Are say hurricanes (as noted above), or say murders in a city, trendable? I guess I wanted to avoid the event type from the discussion and simply look at frequency.

And I did look at a chapter Link about "maximum likelihood estimator (MLE) like a full-blown Kalman filter", but that's a little beyond my pay grade. I hope I don't have to go there.

 

[GregLocock]

I think "number of events per decade" or whatever is a pretty good place to start. EMA will handle that nicely, so would SMA. If you torture the data with some fancy techniques you might find a trend that a simple approach doesn't reveal, but if it doesn't pass some basic sniff tests you'd be hard pushed to justify it.

http://www.aoml.noaa.gov/hrd/tcfaq/E23.html

For example, were there the same number of hurricanes 1900-1950 as 1967-2017? if so then any trend in frequencies is pretty weak or inconsistent. If, as is the case, the answer is that there were fewer hurricanes in later years then torturing the data to find some increase in frequency would need to use some bulletproof method.

I've attached a somehwat cleaned up version of that data, showing decimals of a year and the hurricane strength.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[GregLocock]

Top line is cat 1 and higher, next is cat 2 and higher, and so on (Octave's legend box is cantankerous). The averaging period is 30 years. The greyed out section is the first 30 years. The exponential average is misleading in my opinion because it shows the effect of the zero data right through until 1950. These are trailing averages, that is the date is that of the last year in the average. Normally I'd shift the x axis so that the midpoint is used.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?