Debug, Checkout and Startup

After months of developing processes, creating drawings and integrating components, your automated assembly machine is finally complete. All that needs to be done now is hit the start button, right? Not so fast.

Debug, checkout and startup come first. “Debug” means to search for and eliminate malfunctioning elements. “Checkout” refers to the test of a machine or system for proper functioning. “Startup” is the act of initiating a production process on the plant floor.

Many conflicting objectives must be addressed during the debug, checkout and startup process. For example, a thorough debug at the machine builder might take longer, but it will ultimately reduce problems during startup at the assembly plant.

Even the best engineers know that some type of debugging is inevitable. New assembly and test equipment can be incredibly complex, with countless opportunities for potential error. Process failures usually consume more than 90 percent of the debugging effort. This includes inconsistent part transfer and placement, jams and quality rejects.

Because every automation project is unique and every customer requirement is different, the debug process can change from one project to the next. But, there are general rules of thumb that universally apply to machine builders and customers.

“Debug, checkout and startup are the most important parts of any automation project,” claims Sean Dotson, PE, former president and chief technology officer of systems integrator RND Automation. “I’ve seen some companies that cut corners, thinking everything is good enough just so they can get a machine out the door on time. Saying ‘we’ll fix it at the customer’s site’ is a recipe for disaster. A machine like that will never be 100 percent correct.

“Debug is what gets you to the factory acceptance test,” explains Dotson. “If a machine works at that stage, then it should work when it arrives on the floor of the customer’s facility.

“A customer will typically forgive you for being a week late if the machine runs perfect after it arrives,” notes Dotson. “The alternative is when a machine shows up on time or early, but then requires three or four weeks to fix a problem. Customers always expect a piece of equipment to work once it hits their production floor.”

“The root of debug, startup and checkup is confirming that design meets actual,” adds Andrew Harris, senior instrumentation and controls engineer at ACS Inc. “It’s the process of making sure that the final product meets all the needs and expectations of the customer.

“You never just walk up to a new piece of equipment, load your software and press ‘go,’” claims Harris. “In a large system, there’s always going to be at least one or two devices that give you trouble. Some equipment issues you can work through quickly. However, something small or simple, like a fan or a pump, can cause a big hiccup. It might take as long to fix one basic issue as it takes to confirm 16 other devices.

“It’s rare for a project to always go smoothly,” warns Harris. “Typically, there’s at least one or two things that go wrong. You definitely can’t skip the debug and checkout process and go right to startup.

“Sometimes, there’s premature product failure,” says Harris. “And, just because the instrumentation looks fine, that doesn’t mean everything is necessarily OK. Several years ago, we had a device where a valve wasn’t working properly. We thought it was, because the valve actuator and limit switches worked, but the coupling to the valve body was completely stripped. It was causing a lot of weird diagnostics to occur.”

“Whenever you’re dealing with complex, custom-built automated equipment, there are always challenges,” laments Jason Schwarz, controls engineering manager at BBS Automation Chicago Inc. “Bugs typically crop up at different times. That’s why we assign a project manager who’s closely involved with every step along the way.

“Debug starts once a machine powers up,” explains Schwarz. “It usually impacts the triple constraints of scope, schedule and cost more than any other phase of an automation project.”

“Debug refers to the process of making a machine or system function according to the user requirements,” adds Rob Willey, manager of project management at IMA Automation North America. “Unfortunately, it’s something that has to be done by everyone.

“If everything always worked perfectly, debug wouldn’t be required,” says Willey. “However, that’s not the world we live in. It’s never going to work like that.”

Debugging Complexity

There will inevitably be some mistakes made in building or programming a piece of assembly and test equipment. That’s why debugging is more important than ever. The process has also become more complex than it was it the past, because there’s a lot of more electronics and digital controls used today.

But, the No. 1 goal of all machine builders and systems integrators is to ensure that the debug process goes as smoothly as possible.

“Without a structured method to do IO and functionality checks for subsystems and integrated systems, pieces of the machine functionality can be missed, causing further errors and issues later on in the debug process,” says Brian Romano, director of technology development at machine builder Arthur G. Russell Co. (AGR).

“All the successful engineers that I have worked with have some method of taking notes and [using] checkboxes or to-do items so that things don’t slip between the cracks during debug,” Romano points out. “I have instituted a standard version of this so that I can have any engineer or technician step into a project and understand the functionality, the work remaining and what has been checked, and any anomalies or issues that need to be addressed.”

According to Romano, the debug process can be more complex and time-consuming than many novice engineers expect. “You don’t know what you don’t know until you discover it,” he points out. “There are so many layers to an automated machine, many of which are based on best assessments and design assumptions, both mechanically and electrically.

“You have to design a program to emulate the proposed functionality to animate an idea, which sometimes does not pan out, but you have spent a great deal of time trying to get it to work,” notes Romano. “It was an idea that worked on paper, but maybe it requires mechanical or programming tweaks to make it work.

“The one area that is often asked for, and is not in the best interest of the automated system, is to program around flaws found in other aspects of the machine,” warns Romano. “Hiding flaws behind programming is a house of cards. It is our mandate to not do it that way and to have discussions with all the stakeholders to make the station or machine run the way it needs to.”

“The debug, checkout and startup process is important, because it tests the functionality of the program alongside the physical sensors and mechanisms of the machine,” adds Eric Carlson, chief control systems engineer at AGR. “Developing a functionality check to ensure the program works as expected with a variety of test cases that could occur on the machine is key to ensure the design is robust.

“During the process, you have be able to determine if it is a program issue, a mechanical issue, a physical issue or an electrical issue, and figuring out [the best] solution is not straight forward,” notes Carlson.

Time Constraints

The debug process can be stressful, but also rewarding, for engineers. And, depending on the industry, they can be subject to different amounts of pressure.

“There are always surprises,” says Schwarz. “It’s a questionable and unpredictable process. It’s very rare that we ever have smooth sailing during the debug phase. You have to expect a few curve balls. That can stress out some people, especially junior engineers who don’t have as much experience.”

Because many manufacturers today want to get new products to market quicker than ever to beat competitors, they often want machine builders and systems integrators to work faster. That’s especially true in growth industries, such as any production equipment related to lithium-ion batteries and electric vehicle manufacturing.

“In the automotive industry, there’s often more pressure to ‘get the equipment here and get it up and running fast,” says Willey. “On the medical side, however, that doesn’t happen, because you have to run a lot of product through your machine before you get to a factory acceptance test. It’s nothing for us to run more than 10,000 pieces through a new medical device assembly system, because everything has to get validated.”

Some customers are becoming more demanding with their schedules. Unfortunately, because debug, checkout and startup is the last step in an automation project, it tends to get compressed the most.

“The debug process can be stressful if you’re on a tight timeline,” warns Harris. “Sometimes, engineers are tempted to take shortcuts, especially when they’re faced with a time crunch. However, the shortest route often isn’t the best route.

“Taking shortcuts during debug, checkout and startup won’t help machine builders or their customers,” claims Harris. “It tends to burn you. Whenever they say ‘I don’t need to check that’ or ‘it should be easy,’ many engineers end up regretting that.”

Simulation tools, such as digital twins, have recently helped engineers address traditional time constraints. Many machine builders are now using the technology to reduce costs, eliminate risks and help automation projects run smoother. In fact, many customers now require digital twins.

A digital twin allows mechanical, electrical and controls engineers to work in parallel during the design and development process. They can test things prior to building a physical machine, which results in a reduction in errors, scrap and rework.

“To run off conveyors, robots and other components, we used to have to do it in person,” says Harris. “Now, we can often do it remotely using a digital twin. If it’s accurate, there shouldn’t be too many changes needed later in the process.

“We use digital twins before we start physically building an assembly line or piece of test equipment,” Harris points out. “Once the customer sees the digital twin and approves it, then we start building. It reduces waste long before the debugging process begins.

“We don’t need to have physical equipment to start debugging,” explains Harris. “When the digital twin matches what we’re actually going to be building, we’re able to load software in and confirm that everything works before we even see the equipment.”

“Simulation has made the debug process more streamlined,” adds IMA’s Willey. “Typically, when the software gets completed, we’re still in the build process. With digital twins, we can start testing the code and make sure that the logic is developed. We can check out about 85 percent of the software before it ever gets to the machine. We’re also starting to test HMIs using virtual reality technology.”

“Digital twins enable engineers to verify the design and functionality of a machine, such as critical connections or transfer points,” says Dotson. “However, it doesn’t eliminate the need to conduct traditional debug, checkout and startup steps, like a factory acceptance test.

“Some things are hard to simulate, such as feeding sticky or finicky parts—a rubber O-ring vs. a metal disk or diaphragms vs. springs, for instance,” notes Dotson. “In the real world, friction, reaction time, blockages and jams can be difficult to predict.”

Conflicting Objectives

Many conflicting objectives must be dealt with during the debug, checkout and startup process. That can make the experience stressful, but also rewarding.

“With a fixed endpoint in a schedule, and as the disciplines and labor efforts ahead of us are sometimes late, that compresses our schedule and now the ‘ball is passed to us’ and we have a compressed amount of time to do the startup,” explains AGR’s Romano. “And, as all the labor efforts up front are now complete, [people] sit watching us do our thing, waiting for the machine to run.

“Having multiple people sitting and staring while you work is unnerving and, add the compressed time schedule, and you have a pressure cooker,” warns Romano. “But, when you push the button and the machine does what it was intended to do, there is no better feeling in the world.

“Everyone’s design, assembly and debug efforts come to life to produce a product that is often a one-of-a-kind device or machine,” says Romano. “This is very rewarding.

“However, because there are often multiple disciplines—electrical, mechanical, pneumatic, hydraulic and process—that all have a hand in startup, often there are conflicting priorities, conflicting timetables and a lack of communication of needs across all of the labor efforts,” notes Romano.

“This is an important topic and can make or break a project,” says Romano. “[We have] spent time in deep consideration of this topic and we are currently developing an advanced training method for debug teams.”

The debug process typically begins when all machine building is complete and can take anywhere from days to weeks, depending on the size and scale of an automation project.

“If it’s the first time through, you need to always expect the unexpected,” says IMA’s Willey. “But, if it’s the third time through, the process is a lot more predictable and standard.”

“Everyone needs to be on the same page about what debug is intended to do and what we’re looking for,” adds Paul Beduze, sales manager at IMA Automation. “At the front end of a project, we identify all of the customer requirements, such as parts per minute or parts per hour, and put them into a compliance matrix.”

“Debug typically starts when the mechanical and electrical assembly is completed,” explains Willey. “It’s the ‘power on’ phase. The first thing we do is get the network up and running, checking IP addresses, HMI and firmware.

“Once we work our way through the entire machine, we run the auto cycle,” says Willey. “We run single-cycle at the station level, picking a quantity of parts that we want to see built without a problem. For instance, we’ll assemble 50 parts at the first station before we move to the second station. We make sure that various functions, such as vision systems and leak testers, work correctly.

“After we get through all the stations, we run the system with some type of conveyance device,” Willey points out. “Whether it’s an indexer, a linear motor conveyor or a power-and-free conveyor, we integrate the transport and then start running the machine on auto cycle. We then check for purge and other things.

“Next, we do some overall equipment efficiency runs to see if we can meet the customer’s criteria,” concludes Willey. “Typically, we start with a 15- or 30-minute run, log it, and make any necessary changes or fixes. Then, over the next day or two, we repeat the process over and over again.”

“During debug, what you’re going to run into is always questionable and unpredictable,” adds BBS Automation’s Schwarz. “For instance, there could be some type of workmanship issue with the build. There can be design issues that require rework. There can also be issues caused by things such as loose cables or fittings that interfere with a robotic arm. However, overall, there are three big challenges that we typically face during debug: cycle time, data tracking and fault recovery.

“Commissioning is the first part of our debug, checkout and startup process,” says Schwarz. “We use binders that contain a series of checklists that cover 10 primary tasks. The main objective is to flush out workmanship issues and defective devices. We also make sure that all manual functions work.

“I refer to commissioning as ‘breathing life into the system,’” explains Schwarz. “It involves CPUs and anything that requires codes or an IP address to communicate. Whenever we have a smart tool, such as a robot, press or camera, we have to identify what type of firmware is used and make any necessary upgrades.

“One of the first things that we do is verify the safety IO,” notes Schwarz. “We test all safety functions, such as light curtains, e-stops and guard doors.

“We also make sure that all actuators, sensors, servos and thresholds are set up,” adds Schwarz. “We ensure that all robotic manipulators can operate in the same work envelope at the same time and avoid collisions.

“For anything that requires machine vision technology, we establish things such as camera focus and optimal lighting,” says Schwarz. “We calibrate measurement systems and get all other field devices configured and mapped to the PLC. Finally, we use the customer’s product to make sure that fixtures and grippers fit, and that all mechanical alignments can take place.”

FAT vs. SAT

After debugging is complete, the next step is a factory acceptance test (FAT) to demonstrate performance to customers. This important phase of an automation project includes the customer’s project manager and its support team.

By thoroughly testing equipment and components before they get shipped to the customer, machine builders improve the odds that new installations will perform as expected. Planning, documenting and testing the equipment before delivery ensures that the product will consistently meet the customer’s expectations.

“FAT involves running the machine as if it’s a production-ready tool,” says Schwarz. “We verify everything and make sure that it’s exactly how it will be when set up on our customer’s factory floor.”

The acceptance test helps assure both parties that the automation system complies with all contractual specifications. It also addresses any functional issues before a machine or system is shipped. Rectifying issues while the system is still with the manufacturer is preferable to addressing problems post-deployment.

The FAT covers machining, metal fabrication, paint and finish, wiring and leak testing. It also verifies the functional performance of all mechanical, electrical and control systems prior to shipment. A FAT often includes sample run off and training of staff.

Harris suggests using an instrumentation index to document and confirm that the equipment works correctly. A final acceptance document would be a run-off verifying the performance of the system.

An acceptance test log should also include things such as total time of test, net time of test, run time, machine speed, total output, rejects, good output, chargeable downtime and nonchargeable downtime.

“To ensure smooth FATs, communication and documentation is important,” says Dotson. “The relationship between machine builder and customer should be cooperative, not adversarial.

“Typically, a mixture of people with different engineering skills will work the best,” notes Dotson. “If possible, it’s always a good idea to get operators, maintenance and safety staff involved with the FAT.

“At a minimum, customers should expect to spend a week on the FAT process,” Dotson points out. “It depends on the size, scope and complexity of the project.

“A FAT on a moderately complex machine could be done in three long days,” claims Dotson. “I’ve seen FATs on small machines take one day and FATs on large machines take more than two weeks.

“Typically, the more regulations and rules that govern an industry, the long the FAT will take,” says Dotson. “That’s always the case with medical devices and defense products, which have to comply with various FDA and DoD requirements.

“Once you reach the FAT stage, it’s a lot more expensive to fix something,” warns Dotson. “I’ve seen parts show up that don’t look anything like what was sent for earlier design reviews or that are made from a different material compound. I’ve also seen labels show up in different positions on a bottle or a part.”

After FAT, machine builders disassemble equipment for shipment. Sometimes, it will take more than 10 semitrailers to ship a large system. Other machines will fit in the back of a cargo van.

“We do everything modularly when it comes to mechanical, electrical and software components,” says Schwarz. “That makes the disassembly and assembly process much easier than it was in the past. All air, power and network connections are distributed.

“Once a system arrives at the customer’s factory, our engineering team sets it up,” explains Schwarz. “However, the customer typically does the rigging first, which involves unloading, lifting and moving the equipment into position.”

After the system is delivered, installed and ready to go, a site acceptance test (SAT) is conducted. This is the only way to truly determine whether a piece of new equipment will function as needed.

“The SAT provides an opportunity for final confirmation that the performance experienced during the FAT is repeated after the systems are installed onsite, ensuring nothing has changed or was damaged during shipment and installation,” says Harris.

“This process typically involves full functional testing of the equipment after it is installed and integrated with support systems,” explains Harris. “The same engineers who designed the systems and performed the FAT also lead the SAT to ensure continuity and completeness.”

The SAT provides all parties final confirmation that the systems have met the following performance requirements:
    • Installation is complete and adequate.
    • Integration with supporting systems is appropriate.
    • Important process parameters are achieved.
    • Functionality is aligned with requirements.
    • Usability is intuitive and high quality.

Mistakes to Avoid

Each machine or system creates a new experience for engineers. Ideally, the debug, checkout and startup process should end with a win-win for both machine builder and customer, with smooth running equipment that meets or exceeds initial targets and goals. However, several things can cause problems or delays.

“One of the biggest mistakes that I’ve seen in the past is poor documentation,” says Harris. “Debug, checkout and startup should be a well-documented process. All companies should have a written procedure for how engineers execute it.

“At the start of every project, you should know what your customer’s critical performance values are,” explains Harris. “What is their acceptance criteria and what does their success look like? Acceptance values and variables should define things such as, ‘the machine needs to produce X parts per hour’ or ‘this equipment needs to be capable of meeting a takt time of Y.’ We always like to ask ‘what does success look like?’

“If your acceptance criteria looks different than your customer’s definition, it’s going to be a problem,” warns Harris.

“Don’t take shortcuts and don’t hide design or assembly flaws behind programming,” adds AGR’s Romano. “Standardize design structure so that programs and designs are predictable on how they are executed.

“There is always going to be new ground broached, and that is expected and required,” says Romano. “But, the approach on the layout and steps taken during the design help put it into a position where others that are accustomed to the structure and methods can be brought up to speed quickly.

“Interdisciplinary communication is also a must,” claims Romano. “Daily engineering gembas [should] discuss ongoing designs and get input from others. Gembas can set the direction and focus of each day’s debug, checkout and startup efforts.”