Checking of calculations in the aircraft industry 14

[Andries]

Hi All,

There is an interesting thread under Structural Engineering regarding checking of calculations; some of the replies are quite scary (think Hyatt Regency Walkway 1981).
How do you go about checking of calculations in the aerospace industry?

Andries

 

[edbgtr]

Andries, thank you for raising this issue.

From the responses one realises that the problem has many aspects to it. I’m sure the problems being highlighted on this thread are being experienced worldwide.

I’d like to relate the situation experienced by me with regard to analysis report checking and what I call “gate-keeping”, or lack thereof, of company methods and processes being used to substantiate structural designs. It would appear that those checking other engineers’ work do not have sufficient knowledge of “what” they should be seeing. There is no point in checking another person’s numerical accuracy if the checker doesn’t know how the analysis should be done in the first place.

There are 4 signatures on final certification reports. They are: - (1) original analyst, (2) peer checker, (3) lead stress engineer and (4) department manager. Only the peer checker does a half-decent job. If he’s not experienced enough, mistakes and application of incorrect methods are not picked up. The lead “trusts” the peer checker and the manager “trusts” those below him to have “caught” any bugs/errors there may be in the analysis.

The company has been around a long time and are still providing daily repair support to their products that have been in service for 30 years or more. These old structural analysis reports contain stress methods that are clearly origin-referenced and any deviation from the standard is explained and justified. Needless to say they are all hand written and where early computer methods were used, a complete set of printouts necessary to replicate the calculations is provided.

However, a change has occurred with the advent of work-station automatic stress analysis methods and primarily the use of VB-based spreadsheets. Every component now gets its own dedicated multi-worksheet workbook that produces MS values at predetermined critical areas of the component. Most often that spreadsheet can only be “run” by the creator/analyst. It rarely is accompanied by an explanatory “front sheet”. The checkers therefore have to be familiar with the workings of the workbook or it can’t be checked. The problem is aggravated when these spreadsheet/report combinations are handed over to MRB/sustaining.

The company has its own structures manual, which is an excellent reference document based on classical analysis theory and each subject entry is backed up by a report outlining the theory which it is based on and, where appropriate, laboratory test reports.

The fly in the ointment occurs however, in that many of the stress engineers don’t know what the structural manual contains and the implications and/or limitations of the methods presented in it. The more complex analysis methods can be performed using online program versions of the method. However, some of these are often blindly used to solve problems that that do not always conform to the original analysis assumptions made. A similar problem exists with the “Roark Formula Grabbers” as I like to call them.

Another project-based, primary assembly item phenomenon called “The Stress Guidelines (or Methodology) Document”, has reared its head. (This form of document is now apparently widely used in the Industry) This document contains the stress methods that shall be used by the stress engineers. These documents often contain questionable methods that do not appear in the public domain literature or the company structures manual and are therefore not covered by in-house supplementary substantiation reports.

As examples amongst others, at a detail level, there are two frequently occurring problem areas that are not captured by checkers. The first is the application of bolt group analysis to joints that do not comply with the assumptions which bolt group analysis is based on. The second is lug analysis that suffers from the same problem.

Bolt Groups: Items that should be calculated as multiple support overhang beams are treated as a bolt groups. The flexibility of members between fasteners in the joint excludes the necessary assumption that fasteners further away from the bolt group centroid carry more load than those close to the centroid. In fact the opposite is most often the case.

Lugs: The in-house method used is based on the widely-used Melcon/Hoblit method, not Ekvall. Stress engineers incorrectly apply force systems that ignore the geometric symmetry definition of the axial and associated transverse in-plane axes. In a recent case a non-symmetrically placed bearing was analysed with the in-house lug analysis tool as if it was symmetrical. Out-of-plane offset moments caused additional stresses at the critical section behind the pin. This resulted in a substantial MS reduction. Three tiers of checking did not pick up the error.

While the above examples have not (yet) resulted in serious failures, margins have been dramatically reduced that will affect MRB and/or sustaining engineering functions down the line.

Are similar problems being experienced elsewhere? How are they being dealt with?

Ed.

 

[Aerodesign]

Hello,

I could have written your (edbgtr) piece myself such are the similarities. I have been working at a EU/NL based firm for some years as a consultant trying to get the stress methods and analysis into line with a major customer (who assemble a/c in Toulouse). Much of the customer's methods for the UK projects are based on familiar BAe practices and procedures and are similar to EU/NL in-house methods. However, rather than just adopting these methods there is this overbearing requirement to make the tools 'idiot proof' - making black boxes or canned tools.

This means the poor (sob) engineers trying to actually learn how to do stuff are confronted by an array of black boxes (KBE is the buzz word - knowledge based tools)into which they blindly plug numbers and read off the MS/RFs. Few of these guys get the chance to actually ensure the result matches the expectation.

In some cases these KBE tools have been used even though the internal assumptions are inappropriate for the problem being analysed. As the KBE tool becomes 'secret' over time, people move around, so unless a thorough effort has been made in documenting the KBE tool it is inevitable it gets used again and again, potentially giving bad results.

In terms of checking and signing off these calcs (black boxes are often locked VB routines within spreadsheets or Mathcad) present a nightmare for the Lead Stress (me). Criticism of taking too much time does occur from management. However, l insist on all data flows being documented and 'front sheet descriptions/scope statements' added. Significant oversight of the methods used is also embedded within the certification dossier.

Despite all of the measures mentioned above a 3rd party auditor would still have significant challenges in understanding all and every assumption, source of input/data in those calcs, etc.

So to answer Andries original question - how do we check Aerospace calcs?

a) The author of the dossier completes a checklist that prompts him/her to include standard and specific data, inputs, method descriptions, etc. The emphasis being on content and traceability of data. It includes embedding examples of calcs into the dossier itself. He prepares and documents all supporting files. He adopts a means of doc configuration (via issue and version number).
b) A third party checker (usually part of the same team but ideally from a different project) works through the dossier and underlying supporting files. He/she utilises a checksheet which categorises the issues found. These are documented in a tabular form.
c) The compiler receives the checksheets from the checker and, after discussion, normally makes the changes required.
d) The checker gets to re-check the items in the checklist to ensure all are adequately covered and/or corrected.
e) The Lead Stress confident of the compiling/checking process to that point then takes a view on the dossier and supporting files. The amount of additional checking (dip checking or backwards checking from an RF/MS) depends on the assessment of the people in the chain before him. A rough guide is that 10%-20% of the content is rechecked/dipped. This can lead to putting the dossier back to the checker if the required quality/content is not in place.
f) Once satisfied as to the content the Lead Stress then approves or authorises the dossier and it is delivered to the client (Chief Engineers office if internal or maybe client/Airbus Cert team). They then go through a similar checksheet checking process with feedback to the Lead Stress as needed.

And that is why aerospace projects burn so many hours. An example being 20,000+ hrs stress work on (part 25 a/c) wing components with an additional 2000-3000 hrs on FEM validation and documentation, 5000 hrs of test justification plus design/ME/PV/MRB support.

I hope this helps answer the original question.

 

[SWComposites]

Ed -

Are similar problems being experienced elsewhere?
>Yes. And its worse when margins are written directly from FE stress contour results.

How are they being dealt with?
>If there are experienced leads on the project, then the problems are (hopefully) minimized. But in many cases the leads don't have the experience, or the time, or are pressured to worry more about schedule/cost over technical correctness. The aerospace structures community has gotten complacent - unfortunately it will take a major structural failure resulting in a severe aircraft accident to wake up the industry.

And if you think the state of the initial stress analysis practice is of concern, you should see what constitutes "analysis" in the liaison (MRB) and repair areas.

 

[Andries]

Hi All,

Many thanks for the valuable replies to my query. Looks like SWComposites has touched on a potential can of worms in his second post!

 

[YoungTurk]

edbgtr's post makes me wonder whether he works the same place as me!

In my case, I'm the one responsible for some of the sins he lists (creating a stress guidelines document, adding VBA to Excel workbooks to automate solutions, etc.), but all with the intent of reducing mathematical/numerical errors. The problem is not that the automated methods exist, but that stress engineers may not understand the assumptions which are made when they use a particular method.

The challenge I see is in training and then later checking to ensure that the analyst understood what assumptions they were making when they employed a particular method or formula and how the real world varies from their model.