Automated Seam Tracking System based on Passive

Monocular Vision for Automated Linear Robotic Welding

Process

Átila Astor Weis Adriano Velasque Werhli Nelson L. Duarte Filho

atilaweis@furg.br werhli@gmail.com dmtnldf@furg.br
Federal University of Rio Grande - FURG
Computer Science Center - C3
Group Automation and Intelligent Robotics - NAUTEC

ABSTRACT there is a new design and changes in the production line

Welding is a very important process in the industrial sce- should be made. This way, flexible solutions for robotic
nario, especially in the shipbuilding industry. Many re- welding processes are necessary. Over the last years, many
searches employ robots to automate different welding pro- researchers have worked in robotized welding process in ship-
cess. This paper proposes a system for automated seam yards [4]. Furthermore, the increasing use of robots in weld-
tracking, based on passive vision, to act as a data feedback ing processes can reduce the contact of the human workers
to the automated robotic welding system, with quality and with hazardous environments.
productivity gains in the process. Different cameras arrange- One alternative is the Bug-O Matic Weaver - a robotic
ments are presented and tested as the vision acquisition sys- system remotely operated, (see Figure 1). It is most used to
tem in a commercial linear welding robot. A physical me- perform linear welding in large thick steel plates. The robot
thod and an algorithm to remove visual noises from images, carries the welding torch freeing the welder from laborious
produced by the GMAW welding process, are presented and and repetitive work. The robot runs on a fixed rail, posi-
the results discussed. tioned in any plane surface and should be, in the best case
scenario, parallel to the welding groove. The operator ad-
justs the welding and robot’s parameters during the welding
CCS Concepts process, exposing the operator to the same hazardous envi-
•Computer systems organization → Robotics; ronments of a regular welder.
Vision sensors are being used in industrial applications
due to the amount of information that can be extracted
Keywords from the images and the noninvasive characteristics [6]. Al-
Computer Vision; Welding Automation; Noise Suppression; though, the welding process produces different visual noises
Robotic Systems; Shipbuilding Industry. that interfere in the image capture and degrade the quality
of the information in those images.
1. INTRODUCTION Considering the limitations of the vision based alterna-
tives in the literature, we propose a system that is based on
Robotized cells are present in different industries. In the a monocular camera and algorithms to remove visual noises
automotive industry, for example, welding robots perform and extract useful information during the welding process.
spot welding with great precision and efficiency. Sensors, The system is able to act as a data feedback to assure the
such as a high-resolution laser, map the area to be welded, process quality and to increase the automation level.
providing very accurate parameters to welding robots and
machines. Complex robotized arrangements are made to
meet requirements of these industries and the cost is diluted
by the amount of products manufactured by the cell on a
mass production line.
Differently than the automotive industry, shipyards are
characterized by the difficulty of the production serializa-
tion, with the result that, for every new ship being built,

Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-
publish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
SAC 2017, April 03-07, 2017, Marrakech, Morocco

c 2017 ACM. ISBN 978-1-4503-4486-9/17/04. . . $15.00
DOI: http://dx.doi.org/10.1145/3019612.3019880
Figure 1: The Vision System test arrangement.

300
 Figure 3: The steps of the proposed method to sup-
press visual noise and extract the groove’s proper-
ties.
Figure 2: Comparison between the different Vision
Systems arrangements tested.
the camera using narrow bandpass filter is fixed on the left,
both at a slight angle from the gap’s center. As the welding
process is not repeatable, this way the three cameras were
2. RELATED WORK AND METHODOLOGY able to record the same welding process, allowing a direct
Cameras have been used before to enhance the robotic comparison between the acquired images.
welding process in [5, 10]. Usually, structured light [9, 2] The Figure 2 shows the differences between the visual sys-
and band-pass filters [2, 8] are used for automated seam tems tested during a Gas Metal Arc Welding (GMAW). As
tracking, showing good results even in the presence of visual can be seen, his type of welding produces a large amount
noises (sparks, fume and radiation). However, they do not of fumes and sparks that affect the groove’s visualization by
deal with the main sources of image noise. Differently from the cameras. These noises make it impracticable to extract
the previous works, we proposed a new visual system to the groove’s properties from images, that is why a simple
allow the visualization of the gap in a real welding condition air blower system was used to test the physical fume ex-
dealing with the main issues: such as spatters, smoke, and traction. This physical system was positioned between the
welding noises near the torch [3]. cameras and the welding torch. This simple fume extrac-
The proposed method to reduce the remaining visual noise tion method showed great results enabling the visualization
from the captured images is based in [1]. The paper pro- of the groove, without interfering in the welding process.
poses that the main idea is that the sparks are high-speed Notoriously, after the fume physical extraction, the images
noise, therefore two consecutive frames do not have the same still have some visual noise caused by the sparks. These vi-
characteristic noise. The mentioned method subtracts two sual noises are detected as lines and affect the extraction of
consecutive frames and blinds out the identified not constant the groove properties, degrading the reliability of the data
parts - the spatters - of the two inputs images, enabling the output. Because of this, a noise suppression method is nec-
seam profile extraction and the automated tracking. essary for it to be possible the extraction of the real environ-
The goal of this proposed system is to allow automated ment characteristics and allow the automated seam tracking.
seam tracking, during the welding, based on passive vision. In order to remove sparks and avoid saturated regions on
Therefore, the first step is to identify the best image acqui- the image, we used a basic image composition technique.
sition system (camera, lens and bandpass filters) that en- After two consecutive frames are acquired they are joined as
abled the seam tracking. To do so, we tested three different in Equation 1. This method works in gray-scale images and
arrangements: (i) CMOS camera with band-pass filter (cen- uses two consecutive frames, where the minimum value is
tered at 630 nm) - DRF; (ii) CMOS camera without filter - selected from each correlated pixel, generating a noise-free
TRS; and (iii) CMOS High Dynamic Range camera - HDR. new image.
The physical arrangement of the system to perform the
image acquisition test and select the best imagery compo-
P outt,i = min(P in(t,i) , P in(t−1,i) ) (1)
nents is shown in Figure 1: highlighted are the Bug-o Matic
Weaver - the linear welding robot used for the tests - and The background information does not change between two
the three aforementioned camera arrangements. consecutive frames. Therefore, the combination of two con-
The three cameras in our experimental setup are man- secutive frames keeps the constant background of the two
ufactured by Basler AG and have the same type of lenses. images and eliminates the moving parts, once that the fume
The cameras were positioned at a distance of 15 cm from the and the sparks appears always brighter than the background.
working piece: the HDR camera’s position is orthogonal to This method is dependent on a continuous lighting source:
the gap, the camera without filter is fixed on the right, and the welding arc.
301
 to eliminate spatters from images and maintain the constant
background, used for automated seam tracking in robotized
welding processes. Different visual acquisition systems ar-
rangements were presented and tested during a GMAW weld-
ing process performed by a linear robotic system. In the
tests, a physical solution for fume extraction was used and
the results showed that the CMOS HDR camera presented
the best results for feature extraction in noised environ-
ments.

5. ADDITIONAL AUTHORS
Luciane Baldassari Soares, Cristiano Rafael Steffens, Ma-
theus Frantz de Faria, Paulo Lilles Jorge Drews-Jr, Sı́lvia
Silva da Costa Botelho, all with: Federal University of Rio
Figure 4: Average of the Detected Lines by LSD in Grande - FURG, Science Computer Center - C3 (email:
different regions of the images. silviacb@furg.br).

6. REFERENCES
Figure 3 shows the proposed visual noise suppression me-
thod: two consecutive frames are the system input; after [1] K. Hang and G. Pritschow. Reducing distortions
the new image composition, a histogram equalization is per- caused by the welding arc in a laser stripe sensor
formed in the image, responsible for enhancing the contrast, system for automated seam tracking. In Industrial
then, the LSD method is performed and the lines are fil- Electronics, 1999. ISIE’99. Proceedings of the IEEE
tered by their angle where, only the vertical lines, (between International Symposium on, volume 2, pages
90◦ ± 5◦ ) were kept. 919–924. IEEE, 1999.
[2] Y. He, Y. Chen, Y. Xu, Y. Huang, and S. Chen.
Autonomous detection of weld seam profiles via a
3. RESULTS model of saliency-based visual attention for robotic
In order to compare differences between the acquired im- arc welding. Journal of Intelligent & Robotic Systems,
ages and to make it possible to decide between the tested pages 1–12, 2015.
systems, we applied a line detection method. Line detec- [3] J. S. Kim, Y. T. Son, H. S. Cho, and K. I. Koh. A
tion is a classic usage of computer vision. We used the LSD robust visual seam tracking system for robotic arc
algorithm, proposed by [7] and implemented in OpenCV li- welding. Mechatronics, 6(2):141–163, 1996.
brary. The number of detected lines in each image by the [4] D. Lee. Robots in the shipbuilding industry. Robotics
LSD method is used as a quality indicator for the images. and Computer-Integrated Manufacturing,
Different regions of the images, at intervals of 100 pixels, 30(5):442–450, 2014.
were analysed and the results are shown in the Figure 4.
[5] H. Shen, T. Lin, S. Chen, and L. Li. Real-time seam
The center of the image is the region less affected by lens
tracking technology of welding robot with visual
distortion and optical aberrations and it is the part of the
sensing. Journal of Intelligent & Robotic Systems,
image that has an average illumination. As closer of the
59(3-4):283–298, 2010.
welding process, noisier the region of the image gets and
[6] S. Shirmohammadi and A. Ferrero. Camera as the
more lines were detected, on the other hand, the further
instrument: the rising trend of vision based
away, darker the region and fewer lines in average are de-
measurement. IEEE Instrumentation & Measurement
tected. As can be seen, in the arrangement with the CMOS
Magazine, 17(3):41–47, 2014.
HDR camera, was possible to detect more lines in the center
of the image: in average 7.45 detected lines per frame, com- [7] R. G. von Gioi, J. Jakubowicz, J.-M. Morel, and
pared with 3.67 lines in the TRS camera and 1.75 with the G. Randall. Lsd: a line segment detector. Image
band-pass filter. After this, the combination of HDR camera Processing On Line, 2:35–55, 2012.
with the bandpass filter was tested, but the results did not [8] S. Wei, M. Kong, T. Lin, and S. Chen. Autonomous
show many differences, that is why, the HDR camera with seam acquisition and tracking for robotic welding
regular lenses was selected as vision system. based on passive vision. In Robotic Welding,
After the remaining noise suppression by the proposed me- Intelligence and Automation, pages 41–48. Springer,
thod and with both upper bevel lines extracted, the center 2011.
line can be calculated and thereby, the path to be performed [9] Z. Xiao. Research on a trilines laser vision sensor for
by the welding torch is obtained. The proposed method seam tracking in welding. In Robotic Welding,
helped to reduce the amount of detected lines that did not Intelligence and Automation, pages 139–144. Springer,
match with the actual gap lines, this lines could degrade the 2011.
information gathered from the groove, if done without the [10] L. Zhou, T. Lin, and S.-B. Chen. Autonomous
method, providing inaccurate data for the autonomous path acquisition of seam coordinates for arc welding robot
planning module. based on visual servoing. Journal of Intelligent and
Robotic Systems, 47(3):239–255, 2006.
4. CONCLUSION
This paper proposed a visual noise suppression was able
302