How to replicate a mongodb pod with its persistent storage

https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes

  MongoDB database nodes are stateful. In the event that a container
  fails, and is rescheduled, it's undesirable for the data to be lost
  (it could be recovered from other nodes in the replica set, but that
  takes time). To solve this, features such as the Volume abstraction in
  Kubernetes can be used to map what would otherwise be an ephemeral
  MongoDB data directory in the container to a persistent location where
  the data survives container failure and rescheduling.

Like @Tensibai indicated the issue is related to the replication. According to this blog it could be solved as follows.

 apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "fast"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi


Doing map and filter of a list of dictionaries in ansible

There's a lot that you can do, but I'm not sure with the existing filters you easily augment a dict.

Creating a new filter to do it was easier.

$ cat filter_plugins/addhomedir.py
class FilterModule:
    @staticmethod
    def add_home(_val):
        return [dict(user, home='/home/{0}/'.format(user['name'])) for user in
                _val if 'sudo' not in user]

    def filters(self):
        return {
            'add_homedir': self.add_home
        }
$ cat test1.yml
---
- hosts: localhost
  vars:
     users:
         - name: alice
           sudo: yes
         - name: bob
         - name: charlie
  tasks:
      - debug:
          msg: "{{ users | add_homedir | list }}"

You can use set_facts and loop to get the values you want, but the question becomes what format and use of the data is really needed.  I'm not sure creating a new structure is the Right Way (TM).  It could be easier to just calculate the individual value for each item in the list when it is used.

$ cat test2.yml
---
- hosts: localhost
  vars:
      users:
        - name: alice
          sudo: yes
        - name: bob
        - name: charlie
  tasks:
     - user:
         name: "{{ item['name'] }}"
         create_home: true
         home: "{{ '/home/{}/'.format(item['name'] }}"
       when: "'sudo' not in item"
       loop: "{{ users }}"


How can I find the version of Docker I am using?

For the daemon's version (meaning the engine's version) that's running on your server:
$ dockerd --version
Docker version 20.10.2, build 8891c58

For your client's version that's running on your local computer and with which you send commands to the daemon on the remote server:
$ docker --version
Docker version 19.03.13, build 4484c46d9d


How to execute an ansible playbook using a jenkins pipeline script?

If you have Jenkins user public key, on your target hosts authorized keys file, for the user you want to login as, ssh connection authorization should work.

This is an example of Jenkinsfile step.

ansiblePlaybook colorized: true, installation: 'Ansible', inventory: 'inventory', playbook: 'playbook.yml',tags: "" , skippedTags: "", extras: "--extra-vars ' hst=targetHost "

This is Ansible Inventory example. You can define the args at the host level as well.

[targetHost]
17.14.69.21 ansible_ssh_user=sshuser

[all:vars]
ansible_connection=ssh
ansible_ssh_common_args='-o StrictHostKeyChecking=no -oHostKeyAlgorithms=+ssh-dss'


There are a few ways to approach deleting unused resources, perhaps tagging items that are part of some trusted or known group, as some have suggested here and then manually monitoring for resources that isn’t tagged appropriately.

You could then extend this to a python script that you run every so often to ID and delete these resources.

However, there are other (potentially less intensive) options.

1) AWS already provide a way for us to monitor our resources, identify unused ones and terminate them with CloudWatch alarms. You can start by configuring some alarming based on some set of checks that you might already be doing manually. Once you are comfortable that the alarming is accurate, you can add a termination step to the chain.

2) Alternatively, our friends at Netflix made this chill tool called JanitorMonkey to monitor and delete unused Instances, EBS Volumes, EBS Snapshots and Autoscaling groups. It is part of the original so-called “Simian Army” open source pack from Netflix and written in Java. Even thought it’s a few years old now, this thing is highly configurable and well worth a look.

Both options here allow you to configure warnings and business rules that fit your needs. Additionally, using CloudWatch might also be a good opportunity to learn more about AWS logs and anomaly identification.


Should I use multiple beats input in Logstash?

So still not benchmarked much, but after a quick code inspection and a bunch of reading about netty and logstash in version 5 the input is not the bottleneck to worry about.

Logstash team did put a bunch of work in the way the filters and outputs plugins are run in parallel, the beats input plugin wait for a batch of events, and the performances problem have indeed been solved in version 3.1.0 by the rewrite I quoted in the question.


Application version and Database Schema version mapping

Honestly, you have several questions to unwrap and there is no perfect guidance.  Database versioning is difficult at best.  However, there is guidance that I have been successful with.  In all cases, starting where you are as "version 0" is your best bet.

For your first statement, use one of the following methodologies (or a combination that works best):

EntityFramework Code First Migrations - Treat the current state of your schema as well as any reference tables as version 0.
DbUp and the DbUp Azure Pipelines tasks - This is a very simple, but effective way to get a "poor-man's" database migration methodology, if you follow some simple rules/conventions.  There are also ways to integrate "running" DbUp migrations as part of the start-up of your application, too, but I've had success executing DbUp migrations via Azure Pipelines releases.
Red Gate Database Lifecycle Management (DLM) Automation Suite - This requires a bit more planning and comes with a cost, but Red Gate's tools integrate well and give you more of a complete DLM picture from integration of SSMS with Git version control, drift detection, Azure Pipelines tasks, and more.

As far as the question of should you couple versioning of the database with the application, you'll need to answer this question.  Is the database shared between more than one application or is it dedicated to that application.  If "dedicated" is your answer, then versioning of the database with the application is the right thing to do and should be done so within the same source code repository.  If not, it will be difficult to decide which application "owns" the versioning of the database or if it needs to be versioned entirely independent of your apps.  If it is versioned independent of your apps, then you better have strong unit testing in all of the apps.

Lastly, I will assume your "pitfalls of using such logic" refers to using your applications initialization/startup code to run an upgrade of the database to the intended version.  If I've assumed correctly, there is no right/wrong way.  What matters is that you understand where you need the automated process to fail to ensure your application does not incur a downtime because of a failed database migration.  If you upgrade a DB during initialization/startup of an application, then your app better have a fallback mechanism where it can function with the previous version of the DB until which time it's fixed.  This is where I'd recommend making failure of a DB upgrade part of the deployment process itself and not part of the application.  That way, if the DB upgrade fails (and you're doing it correctly), reverting to the previous version of the app upon failure ensures that downtime is minimized.

And, when I refer to doing DB upgrades "the right way," I refer to using methods that ensure the state of the DB schema is only changed atomically and transactionally such that failures in the middle of a deployment can be completely rolled back.


Conflicting Kubernetes CPU usage & Docker Container Metrics

Multiple things here:

Youre on AWS ec2, hence whatever you are doing on your instance for measuring CPU is calculating CPU on hypervisor level and not instance level. To very this run any load test and check iostat -ct 1 and CPU usage in cloudwatch. CPU usage in cloudwatch is always 10-20% greater than what iostat will report and that is because iostat will give CPU usage on hypervisor level.
As docker to see how kubernetes and docker metrics compare i suggest to run the containers with --cpuset=1 or any number to allow all containers to use only a single vCPU.
Also in AWS 1 CPU= 2vcpu. It is hyperthreaded to 2. You can maybe take this into consideration while calculating.
Finally the best metric to use to see CPU usage for a particular application is using htop and correlating with your cloudwatch metrics.
I have also observed sometimes docker daemon pinning itself to one of the virtual CPUs and hence when you are reducing it to 1000m, maybe the whole setup is getting slow because the reduction is happening on any one of the vpcus. You can use mpstat to get into details of this.
Lastly on the host level you can pin docker to a single CPU and observe more.

Hope this brings you somewhat closer. Update me if you have already found a solution.


How to deploy a ZIP or WAR equivalent (built locally via npm)?

There are now two possible solutions: One NPM only one, and a clean one.

First the NPM One (where im not quite sure what exactly will work):

I just googled "npm deployment" and found 3 different Plugins which do exactly that what you want ( 1, 2, 3) and there are many more. Which now you suits you best, will be your choice.

Now the cleaner one (because we are in the DevOps Stackexchange ;) ):

Use a Dedicated Service (self-hosted or provided) to build and deploy your application.
This way you will just commit your sourcecode into the VCS and the build system will build your Artifact. Then you deploy it through there via it's possibility for it.

As Example i used Atlassian Bamboo quite often for that.

I commited my code
Bamboo gets a Trigger to build
Bamboo run the Build with the Tests
Bamboo create the Artifact
now you had the Choice to automatically deploy the Artifact or wait for a manuell Button press inside the UI

if automatically it will start a Deploy Task
else on Button Press

The Deploy Task is the configuration how it will be Deployed.

in your case: copy the ZIP, extract it, run the deployment.sh

One thing beside that all. One thing i would never do is to build ON the Live system. Sourcecode on the Live Server (even Javascript one, which is most of the time minified) is a big attack window. Beside that you could get many weird behavior due weird things that the different systems do inside the building of the artifact (like different dependencies or things like that)


Zero Downtime Deployment with Database Backward Compatibility

There is plenty of information about this. The basic idea goes like so -

Create a script that does the database migration, but make sure that it only adds tables and fields - never removes, renames, or changes the type of existing tables or fields.

The new version of the application will start using the new tables/fields when it is an existing field from the previous version that needed a change - the app must continue the previous version as it used to, and start using the new version as well.

In some cases, before the newly created field can be used it needs to be populated based on the previous field - the application can do this when it first starts.

If some functionality depends on the new field, and it is not yet ready, opening it to users can be hidden behind a feature flag that is toggled when ready to expose the new functionality.

When yet another version is released, the application can finally stop writing to the old field you were using and start using the new field exclusively.

The above mostly explains what happens when fields need to be renamed or type changed. For new tables or fields, it is much simpler since the existing version is not using these.


How to create ALB that can handle traffic on 2 ports

Add listeners to the ALB

1 listener for 3701
1 listener for 3702

update the ALB action for each Listener to point to the the correct Target Group


DOCKER_CONTENT_TRUST=1 this is env variable to enable Docker content trust by default it is in disable mode.

read more about DCT in official docs DCT docs

For specific  build/push/pull you can use as docker command line flag as well

--disable-content-trust=false or --disable-content-trust=true


As it says, No such file or directory: 'gpg', so you probably need to install gnupg before adding the PPA repository:

apt-get install gnupg


Best way to isolate programming projects?

A more generic approach would be use a separate virtualized environment for each project, see virtualization. That could be either:

a container image - more lightweight, faster and definitely more popular these days, see containers
a virtual machine (VM) image, slower and a bit more heavier to maintain than containers

You'd have all the project dependencies (IDEs, libraries, tools, etc) installed inside the virtualized environment, not on your system itself.

You'd need to create/bring the project's corresponding image on your machine when you start working on it and you can delete or save it for later re-use on some external storage when done. A shared artifact respository would be recommended as storage if more than one developer can work on these projects.

A huge advantage of this approach is that such virtualized environments play really well with automated CI/CD pipelines, in true DevOps spirit, helping to minimize/eliminate the differences between the development and staging/production environments.

Depending on the programming language and/or the supporting development tools being used on a particular project you may have some more specific approaches/alternatives. For example Pipenv or Virtualenv/virtualenv can be used to obtain such isolation for Python projects. But these solutions are typically not usable in other contextes.


You have your VAULT_ADDR set to HTTP, but vault is listening on TLS - the error you are receiving is the HTTP request receiving a TLS handshake.

Try changing VAULT_ADDR to https://mydomain.com.


What's the equivalent of `heroku container:release web` without using heroku-cli?

You can install the Heroku CLI via npm - which might be the simplest.

Note that your Heroku API key has to have the env var name HEROKU_API_KEY for the Heroku CLI to accept it as an auth token (ie. to avoid having to do heroku login).

          - npm install -g heroku

          # build the Docker image (this will use the Dockerfile in the root of the repo)
          - docker build -t $HEROKU_APPLICATION_NAME ./backend
          # authenticate with the Docker Hub registry
          - docker login --username=_ --password $HEROKU_API_KEY registry.heroku.com

          - docker tag $HEROKU_APPLICATION_NAME registry.heroku.com/$HEROKU_APPLICATION_NAME/web
          - docker push registry.heroku.com/$HEROKU_APPLICATION_NAME/web
          - heroku container:release web -a=$HEROKU_APPLICATION_NAME


Continuous Integration Concerns

In short: Docker in Docker (dind) doesn't handle concurrency well.

The reason why you shouldn't use dind for CI is because Docker was designed to have exclusive access to the directory it uses for storage (normally /var/lib/docker). Dind doesn't respect this as all child processes use this directory concurrently. Every time you rebuild (from CI for example), anything related to your app in this directory could get wiped out and forced to start from zero. How would your users like it if they entered their payment details, clicked "Purchase", and suddenly found themselves back on the login screen as if they'd never done anything? That's just not good UX. Two rebuilds occur at once? That's really going to end badly for everyone involved (including your data integrity).

Other Concerns

From the link the OP posted, security concerns arise as the system will try to apply security policies in a very "CSS-like" fashion where a lower container could have access to an outer container's resources unless explicitly prohibited. Remember when you could access web server resources by doing something like "mywebsite.com/../another_folder/private_resource.txt"? Also, sometimes filesystems just don't play well with each other when they're nested in this way.

The Fix

Thankfully, the blog post in the OP has a good solution for the issue. Unless your needs are not met by "build/run/push Docker containers from your CI system itself running on Docker", you can use -v mode (add a data volume to your container) on the Docker socket (usually /var/run/docker.sock:/var/run/docker.sock) to allow the kind of access you need to the "shared" data volume. These containers will be started alongside the parent, instead of underneath, forcing synchronous IO. Now you have the same thing (almost) as dind but without the downsides that come with Docker not being build for concurrency.

Reference (from OP): Using Docker-in-Docker for your CI or testing environment? Think twice.


Is there evidence to suggest that the Docker/Hashicorp ecosystem requires a higher calibre of software development engineer?

I apologize for the novel length response, but I want the industry to start having these discussions as the potential advantages are huge.

The problem is not that it is necessarily more complicated but that it requires experience that is difficult to find today and which goes against traditional models.

This is not an uncommon occurrence in the computing industry, and I personally remember a time when TCP/IP was considered too complicated.  The talent base will grow over time though.

Here are the main barriers I see today:

Tools:

On the platform side most Engineers are just really starting to leverage automation. Unfortunately they are leveraging configuration management tools which were designed to solve a different problem. As an example Puppet and Chef assume idempotency and long lived systems.  A large portion of the industry is just starting to get good at those tools but they are inappropriate for a model where dynamic istanciation is the rule and long lived systems are rare.

Most of the monitoring tools or management tools in use also assume long lived systems do not work well in a dynamic environment.

Then there is the whole we must have a single source of truth problem that I will ignore, however it is a significant barrier.

Virtualization:

This is mostly an issue with vendor lock-in and market forces.  But many SE's have been separated from the simple mechanics of virtualization that their is a knowledge gap.  These individuals are capable of the work but have spent most of their time managing vendors so understanding concepts like linux namespaces or c-groups is almost foreign.

Networking:

Docker and Kubernetes really need network segmentation, and tenant isolation.  While this happens at a google scale most companies are using fragile and expensive enterprise solutions.  These tend to either have no automation or massive centralized management.  Really each swarm or cluster needs it's own segment and the technologies like vxlan or sdn that enable this are just popping up.

Security:

Related to the lack of overlay networks, or shared VPCs between swarms and clusters across security domains this is a very hard nut for companies to crack right now.  Most security teams have no way to figure out how to audit these systems.  Most teams I see are trying to fix it with solutions that don't directly address the problem.  Segmentation and limited super user access will improve security in the long run, but like most humans they can't really parse json policy files with ease.

Related to the knowledge gap in virtualization, most team do not understand the attack surface involved here.  Mix this in with a tenancy to give into the need for super user access this is an unsolved problem for organizations.  As an example the default AWS VPC limit per region pretty much means anyone running kubernetes in AWS today most likely has serious security issues and any breach will be massive.

This is less of a problem on GCE but it is a very real concern today. This is far from an unsolvable, but it is a real problem.

In closing:

With all of those negatives I still think it is one of the most promising directions looking forward.  But our industry today is full of companies that struggle to even implement OpenStack, which is really just a a bunch of python wrappers, a message bus and a database.

It will be a hard sell for a while, but the advantages of true loose coupling will probably win out in the end.


Docker - which cache is used to build image

Answer

The default location of the image cache is under /var/lib/docker.

More answer

The "cache" features in docker are more like a communication vehicle for the documentation, than a proper, separate data structure. There is no special storage which makes up this cache.

For every filesystem-modifying line (RUN/ADD/...) in your Dockerfile, docker run creates a new image, with the image of the previous line (or the FROM line at the beginning) as the base. Docker uses the hashes of the previous image together with the hash of the RUN line to calculate the hash of the new image. If that new hash already exists, then it is reused.

Those intermediate images are exactly the same kind of object in storage as your final image (i.e., a subdirectory of /var/lib/docker, by default), they just don't have a tag pointing to them, so they can be cleaned away somewhat more aggressively by docker image prune -a, for example. But technically they are just the same as tagged images.

All options which talk about caches in the docker documentation (i.e., build --rm=false, build --squashetc.) just modify the behaviour so that said image handling a bit different, but under the hood there's just images, nothing special related to caching.


Every jenkins build shows error: ln builds/lastSuccessfulBuild /root/.jenkins/jobs/Build/lastSuccessful failed

I had a similar issue with Jenkins running under Kubernetes with Azure-storage-backed PersistentVolumes.

Azure storage doesn't support symlinks by default.

As per: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#cannot-create-symbolic-links---ln-failed-to-create-symbolic-link-t-operation-not-supported

To resolve, I added the following to my PersistentVolume yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: azurefile
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  azureFile:
    secretName: azure-secret
    shareName: jenkins
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - mfsymlinks

The mfsymlinks allows symlinks to be created on Azure storage.


why is "docker run" used to connect to redis?

In short, docker exec is used to operate an already-running container, while docker run -it starts a new container interactively.
When you ran docker run -it --network foo --rm redis redis-cli -h rd you created a 2nd container, this time using the redis-cli image, which was used to query your running redis container.
Were you to repeat the steps you took, and then open another terminal and run docker container ls, you would see two running containers.
Reference: https://spin.atomicobject.com/2018/10/04/docker-command-line/


Where is Jenkins Pipeline script stored and can I control it?

The good news, is that you absolutely don't have to use that little box to manage your pipeline script.
Jenkins also allows you to source your Jenkinsfile from an SCM repository like GitHub, and will even let you automagically import branches as separate jobs with separate Jenkinsfiles per branch using multibranch pipelines.
Pipeline scripts are in a language known as Groovy, not Perl or sh or similar. The definitive guides to Groovy are (of course) the Jenkins reference guide and the Groovy reference guide.


Is it possible to have multiple gitlab-ci files in a single repo?

No, you can't have multiple gitlab-ci files per repository.
You can see more information in the following links :

https://gitlab.com/gitlab-org/gitlab-foss/issues/18157
https://gitlab.com/gitlab-org/gitlab-foss/issues/28592

Now, there are some work in progress about this, but it's not available yet.
For example, you can see here ( https://gitlab.com/gitlab-org/gitlab-foss/issues/56214 ) that they are working on pipeline which can include other (complete) pipelines.

The include keyword is more about reducing the size of your main gitlab-ci file.


This question is discussed in some detail in issue 7950  The conclusion seems to be that:

Depending on your storage configuration, the kernel page cache may be used if multiple containers are accessing the same file (e.g., see OverlayFS and Docker Performance).
You may be able to enable Kernel same-page merging through use of of the ksm_preload tool.


There is another manual for enabling reverse lookup DNS in azure.

Will my reverse DNS records resolve from the internet?
Yes. Once you set the reverse DNS property for your Azure service,
Azure manages all the DNS delegations and DNS zones required to ensure
that reverse DNS record resolves for all Internet users.
Are default reverse DNS records created for my Azure services?
No. Reverse DNS is an opt-in feature. No default reverse DNS records
are created if you choose not to configure them.

It seems that reverse dns has to be enabled if it is used internally.


Install and remove a program in the same Puppet declaration

I think that the simplest solution is to have the lifecycle of the 7-Zip package managed by exec resources rather than as package resources.  Imagine an exec resource that installs 7-zip, lets call it install-7-zip.  There is also a second puppet exec resource that uninstalls 7-zip, lets call it uninstall-7-zip.
If all of your other puppet resources that need to use 7-zip then require 'install-7-zip' and notify 'uninstall-7-zip' then they will all be actioned whilst the executable is available.
however its not a very puppet way to solve the problem. It means that even if there are no changes required, you will still be running these exec resources, and puppet will think it has changed the platform.
is repackaging the azure pipeline package a possible alternative? perhaps re-zipping it with a different compression setting? or generating a self extracting 7-zip exe instead?


Auto create Jenkins jobs for branches and pull request via declarative code in scm

Use the Bitbucket Branch Source plugin, which is an official plugin developed by the Jenkins developers.  This plugin automatically creates jobs for all branches and pull requests of all repos in a project, and if you use Bitbucket Cloud it will also set up webhooks in Bitbucket to ensure pushes and PR updates automatically trigger builds in Jenkins.


docker: Error response from daemon: Conflict. The container name "/basexhttp" is already in use by container

You're attempting to create two different containers, each with the same name. Each time you execute docker run it will create a container, and you've named the containers the same with --name basexhttp. These names must be unique, even among stopped containers. To delete an unneeded container, you can stop and remove it with:
docker container stop basexhttp
docker container rm basexhttp

(Note that docker stop and docker rm also work, but it's easier to read and understand the more verbose syntax)


If I set up a Travis CI agent, will it be independent of Travis's existing infrastructure?

Travis CI doesn't offer running your own workers, you use their infrastructure unless you mean the on-prem Travis Enterprise. You might be thinking of GitLab CI?


Is there any motivation to achieve SixSigma or similar in DevOps?

This may sound mean but 6 Sigma does not translate that well to computer systems. The general ideas are valid but the numbers are not. It is intended for the meat world, not a place were we routinely do calculations on gigs or tb's of data a day. Assume you run 1 trillion operations (not much when a regular desktop computer can do ~12 billion a second) at 99.99966% success you are still failing 350k times!

Yes, most of the operations on the code side will actually be made up of multiple CPU operations
Yes, some of this is caught by hardware via ECC etc

What would stand up would be trying to use S6 against the changes you make to a computer system (code pushes or otherwise). Not the operation of existing code (which needs a much higher level of reliability than what you'd use to evaluate PPM on a shop floor for physically created stuff).


Best practice for installing new jenkins server

There is no single "best" practice.  Use whatever approach you and your team feel comfortable with and fits well with the rest of your infrastructure.

If you want to do the Docker approach, you might find this person's writeup worthwhile.

To answer your questions:

  If you were to use a Docker container, I know you can save all the jobs and configurations on external volumes. How would you go about keeping the updated plugins so you won't have to reinstall them if it falls?

Depends.  You may want to write a script or configuration management recipe that installs all your plugins for you and/or configures your jobs.  You can also create a custom Docker image containing Jenkins, all of your plugins, and all of your job configuration.  You can also combine these two approaches, using the aforementioned scripts as part of an automated image build process. Whatever you do, you should definitely ensure whatever files (e.g. Dockerfiles) and build steps (e.g. scripts) are part of the process are stored in configuration management in order to make deploys repeatable.

  Is there a better way than Docker commits/images/saves? Those feel like they could quickly become clunky.

Again, it's really just a matter of what you're comfortable with, there's no such thing as "best"; instead aim for "it does what I need it to do".


How to update pods if version hasn't changed?

The main problem here is, that from a Kubernetes point-of-view nothing has changed to your deployment. So nothing gets updated.

The best solution here would be, that you add something to your image tag, like the git short hash, for example. The your update will just work. Also you will make your deployments re-creatable, when you do it this way.

If you really want to keep it the way it is, you could add some other information to the deployment, that changes on each deployment, but is not used, like an annotation. But I would not recommend that, since I saw this leading to strange behavior with old pods no being shutdown.

On a side note: The imagePullPolicy is just evaluated, when there is change to your deployment. So basically this flag says: If you do deploy something (e.g. bc there is a new deployment), no matter if you have the image already locally, reload it from the registry.


How to pass parameters to a container orchestrated by Kubernetes similar to running the container at the command line using the docker run command?

You can include the arguments in the deployment yaml.  The official docs include examples and further customization.

spec:
   containers:
     image: myimage
     name: myContainerName
     args: ["me", "yes", "https://my.url"]


Gitlab: Always having the possibility to launch a manual job (even if something failed before)

You can use when: on_failure, in your case you can create a new job for rollback when your some more tests have failed.
Example:
cleanup_build_job:
  stage: cleanup_build
  script:
    - cleanup build when failed
  when: on_failure

For more example: https://docs.gitlab.com/ee/ci/yaml/#when


Jenkins Rerun Old Build with Specific Commit

As a short term solution you can replay you last good build. Read the docs for more information

Update: There is a bug , so the proposed way will work only in multibranch pipeline. So as a short term solution you can convert your job to a multibranch pipeline and to use a replay when it’s needed

As a long term solution I would separate build and deployment job and store the build artifact in private npm registry. In that way you can always pick whatever version is needed to deploy


Ansible: Backing up variable value doesn't work as expected

It is a totally expected result. You define the same variables in different places and ansible applies its variable precedence

First of all, remember that variables passed as extra vars always win: you can never change them in any other place or during the run (i.e. set_fact).

Now let's decompose what happens. The schema is basically the same for all vars where you don't get the expected result. I'll only give the example for backup_ansible_port. You can follow the spaghetti yourself for the others once you got it.

When you hit the debug task to display "{{ backup_ansible_port }}"

The first jinja2 template pass finds it should be an expansion of ansible_port as defined in your play vars.
ansible_port is defined with value "11111" directly in your inventory as a host var (see #8 in above precedence doc) But it is also defined as an expansion of tmp_port in your play vars (see #12 in above precedence doc). Higher precedence wins, so the second jinja2 template pass finds it should expand tmp_port
tmp_port has value "22" in your extra vars which will be the final value returned.


Why are you adding the image tag to the pull policy? you dont need it there, just leave it like this: imagePullPolicy: {{ .Values.image.pullPolicy }}


Organizing the build steps on AzureDevOps

You can separate the steps, especially as it would let you do divide the work among different agents. You can also leave them in a single agent if you wish.

The primary advantage I see is leveraging different pools for different sets of work.

With releases last I checked variable scope was limited (as of 2018) to the "phase".

I wouldn't necessarily worry about performance due to connections unless you've observed this issue and have more specific detail you can add.


The Git Server

It's best practice to setup a Git server for the team to access.  This way you have a central server that can be maintained, backed up, and focus on serving the source code.

While you could create your own, there are also several popular commercial options available that often have free private repositories (non-exhaustive list):

Bitbucket
GitHub
AWS CodeCommit

The Git server will just be a repository for your code.  It will allow for workflows like Pull Requests to better manage the changing code base.  However, the Git Server will not build your code or deploy it anywhere.

Jenkins

Jenkins will not actually store your source code.  Instead it is a tool used for Continuous-Integration (CI) and Continuous Deployment (CD), a big part of DevOps culture.  The basic idea is that the Jenkins server will take the source code put into your Git server, ensure it compiles, automated tests pass, and possibly even deploy to your test environment.  Tools like Jenkins work with multiple programming languages.

Maven is a tool is specific to building Java projects.  Tools like Jenkins often have options to use Maven to build the project.

Setting up a CI/CD tool like Jenkins would be a great next step after you have a consistent place for everyone to store their code.

Best Solution

Gitlab, Bitbucket Pipelines, Azure DevOps are just three examples of tools that both house Git repositories and can do CI/CD. There are many options available for Git and CI/CD, and you can even combine them if you choose not to select an all-in-one solution.  Which tools are best is opinion based, and the best practice is to use a tool that helps manage your Git repositories and a tool for CI/CD.


How to set a shared-library globally using the JCAC plugin?

https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/demos/jenkins/jenkins.yaml

It should be possible by configuring it like this:

unclassified:
  globalLibraries:
    libraries:
      - name: "some-lib"
        retriever:
          modernSCM:
            scm:
              git:
                remote: "https://github.com/some-project.git"
                credentialsId: 'some-credentials'


You may be describing a key value store. It is unlikely that you need SQL or relational.
To avoid vendor-lock-in, and to control your costs. Avoid complex services from the cloud providers. Rent a machine or virtual machine from them, and put your own services on it. There are many Free and Open Source databases. Do some research: start with key-value stores (these are the simplest, fastest, but also low functionality, databases).


How to automate installation of Windows installer on Appveyor CI?

The installation of Windows application can be automated using AutoHotkey (AHK), similar as Winetricks does this for all common apps which support.

Here is example of AHK script which aims to install mentioned application:

Run, mt5setup.exe
WinWait, MetaTrader 5 Setup
ControlClick, Button1
Sleep 100
ControlClick, Button3
WinWait, MetaTrader 5 Setup, Installation successfully completed
ControlClick, Button4
Process, Wait, terminal.exe
Process, Close, terminal.exe

To integrate it with CI, AHK needs to be installed by downloading ZIP file, uncompressing it and passing the above AHK script. Suggested appveyor.yml file:

install:
- curl -sLo https://www.metatrader5.com/en/download
- curl -sLo http://www.autohotkey.com/download/AutoHotkey104805.zip
- unzip *.zip
before_test:
- dir
test_script:
- AutoHotkey.exe install.ahk
build: off
platform: x86

where install.ahk is the file to contain above AHK script which can be dynamically generated or downloaded from some repository.


How to deploy to Appengine standard from gitlab CI?

AFAIK there are only 2 ways of deploying GAE apps:

using the gcloud app deploy command:

  gcloud app deploy - deploy the local code and/or configuration of your
  app to App Engine

using the Admin API. From Deploying Your Apps with the Admin API:

  To deploy a version of your app with the Admin API:

  Upload your app's resources to Cloud Storage.
  Create a configuration file that defines your deployment.
  Create and send the HTTP request for deploying your app.

Both gcloud (used in the 1st method) as well as gsutil (used in the 1st step of the 2nd method) are part of the Google Cloud SDK, so you'll need to have the SDK installed on your runner.

Summarizing the required steps from a GAE deployment example using the 1st approach that I found in Getting started with Angular 6, GitLab CI/CD and Google App Engine (probably a good read for details):

in gitlab set env vars with your GAE service account and project ID info
in your .gitlab-ci.yml file's deployment section add a before_script section for installing the SDK (update for current SDK version)

        before_script:
          - wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-218.0.0-linux-x86.tar.gz
          - tar -xzf  google-cloud-sdk-218.0.0-linux-x86.tar.gz
          - ./google-cloud-sdk/install.sh

and a script section for the actual deployment, referencing those env vars:

        script:
          - echo $GAE_KEY_FILE > gae_auth.json
          - ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file gae_auth.json
          - ./google-cloud-sdk/bin/gcloud app deploy --project=$GAE_PROJECT_ID


AWS MSK: deploying all Kafka brokers in the same availability zone

I cannot back it up with links to the documentation, but my strong assumption is that it is not possible. AWS MSK forces cross availability-zone deployments. (In most cases, it is a good idea, anyway. I understand that the scenario in my question is not so common.)

The only option to stay in the same availability zone is to use only one broker. However, it is not recommended to run Kafka with a single broker. Therefore it is not an attractive option.


How to enable debug mode for Docker client on Windows 10?

Eventually, using the docker -D -l debug setting for the client  I have found the log outputs (apparently currently there are no debug level messages implemented as reported.

C:\Users\user\AppData\Local\Docker\log.txt


Terraform - referencing existing security group

You can retrieve security group from data source

variable "security_group_id" {}

data "aws_security_group" "selected" {
  id = "${var.security_group_id}"
}


One could git svn clone a subversion repository and use submodules.


mysqld: [ERROR] Found option without preceding group in config file /root/.my.cnf at line 1!

The default Dockerfile shell is sh. I don't think echo there has the -e modifier for backslash. Newline & other special symbols will work by default. In your command, resulting .my.cnf would contain a litteral -e.
Remove -e or use the original command in Dockerfile in a different shell, for instance:
RUN /bin/bash -c '<your-command>'


How can I get terraforms extern to execute `ssh-keygen -y -f ~/.ssh/id_rsa`?

So what happens in first case is, as Dan's already said, there's no shell used and as such nothing to expand the ~. Quoting the documentation about program:

  Terraform does not execute the program through a shell

On the second case, bash receive in stdin something like this:

{ "-c": "ssh-keygen -y -f ~/.ssh/id_rsa" }

And this looks like a command block for bash, but -c is not a valid command.

What could work could be this kind of program (assuming no specific input):

jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{"pubkey":$pubkey}'

So something like this should work to get the key in local_key["pubkey"] if I understand the documentation properly:

data "external" "local_key" {
  program = [
    "bash", "-c jq -n --arg pubkey \"$(ssh-keygen -y -f ~/.ssh/id_rsa)\" '{\"pubkey\":$pubkey}'"
  ]
}

There's a need to use bash for a one liner to take advantage of command subsitution. You can also do a .sh script like:

#!/bin/sh
jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{\"pubkey\":$pubkey}'

And call this script in the program parameter.


Enable/disable resource in terraform using count

I believe you should be able to reference the namespace name like so:
namespace_name      = azurerm_eventhub_namespace.eventhub[0].name
However, if the namespace_name field is a required parameter on the resources you are passing it to, you might run into other issues while you are trying to set enabled to false, as the resources depend on the output of the resource you are setting to count[0]. If you want them all to be conditional on the creation of the namespace, i would recommend continuing down your approach with the for_each over the local values.
If you need to the other resources to not be created unless the namespace is created you can create a condition on your local value like this (assuming map):
locals{
    hub = var.enable_eh ? {foo = "bar", baz = "qux"} : {}
}

The for_each will omit the resources that are empty.


Terraform plan -var-file="myvars.vars" does not run

You define variables in variables.tf file. However, you can define inputs to these defined variables in a .tfvars(recommended) or any other files.

  Values for the input variables of a root module can be gathered in variable definition files and passed together using the -var-file=FILE option.

  For all files which match terraform.tfvars or *.auto.tfvars present in the current directory, Terraform automatically loads them to populate variables. If the file is located somewhere else, you can pass the path to the file using the -var-file flag. It is recommended to name such files with names ending in .tfvars.
  more...

Make sure you are running terraform plan from the directory where the variable file resides.


Instead of writing your own Dockerfile, you can simply pass auth through environment variables to official images, I use compose file something like this:

version: '3.1'

services:
  mongo:
    image: mongo
    restart: always
    container_name: mongo
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: abcd
    ports:
      - "27017:27017"
    volumes:
      - ./databases:/data/db

  mongo-express:
    image: mongo-express
    restart: always
    container_name: mongo-express
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: abcd

you can have your database persisted in ./databases folder and nice in browser ui at port :8081


Azure Devops, multiple on-prem environments, variables per environment

You can create a Release Pipeline with a stage for every environment.
On each stage you can copy files, etc. and run tests against the environment.
If the test/stage succeeds, the pipeline goes ahead to the next stage.
Between the stages, you can configure that an authorized person has to approve the next step.
The screenshot below is a release pipeline of one of my little side projects.


Is Continuous Delivery the Control Framework for DevOps?

In my opinion you are right, Continuous Delivery (CD) is not the control framework of Devops, at least it is not the only possible one.

But in the context of the book you're quoting it gets the most used possibility when you start to include security baseline and assessments as part of the product delivered.

In a security context, you'll add smoke tests in a post deploy phase to ensure your application is not subject to XSS or Sql injection for example. If any of this tests fail, block any deploy in other environment, mark the deploy as failed and eventually rollback to previous version.
This enforce the security rules to be satisfied, and in this way act as a control framework to ensure production deploy are 'safe' for known vulnerabilities.


how to send a password to MySQL with docker?

As the documentation clearly states docker run command should be invoked as follows:
docker run [OPTIONS] IMAGE[:TAG|@DIGEST] [COMMAND] [ARG...]

so when you run
docker run mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password

Docker assumes -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password is a command you want to run inside the container, not a list of options for docker run command.
The proper docker run invocation with the parameters you specified is:
docker run -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql


A value stream is a process that adds value to a collection of things that have less inherent value.  The canonical example there is an assembly line.  I have little interest in a disconnected collection of parts (steering wheel, shifter, gas pedal.)  On the other hand, I'll pay good money for a car.  Car manufacturers add value to the parts in the form of facilities, skilled work, supply chains, etc.  Parts go in one end of the stream, cars come out the other.

Enterprises use complex processes to deliver their products and/or services.  Value stream mapping is a process used by companies to figure out how and where they deliver value.  The purpose of value stream mapping is to develop a clear understanding of the steps, time, and resources required to support a business capability that delivers value.

Value stream mapping is useful in kaizen, or Continuous Improvement.  The "systems thinking" employed in DevOps and Lean encourages small, incremental improvements made throughout a complex process.  A value stream map can expose areas of waste, such as unnecessary or inefficient steps in a value stream.  A complete map tends to expose those areas of waste.  This helps prioritize improvements.

Value Stream Mapping can be applied at large scope to a business process, or to a smaller scope within software delivery.  The software delivery process itself mirrors an assembly line.  Code is written, reviewed, tested, integrated, and tested again.  It is delivered from one environment to another (in manufacturing this is called a work center) until it is finally delivered to Production.


I'm posting this here not because I endorse these solutions (in fact, I've never tried them), but just because they are a potential answer to your question:

You can start with JenkinsPipelineUnit, a unit testing framework for Pipeline scripts.

There is also a project called jenkinsfile-runner which executes your Jenkinsfile in a transient, headless Jenkins instance.  Supposedly, this can be used for integration testing Jenkinsfiles and Pipeline shared libraries.  However, as of summer 2018 there is no documentation on how to use this tool for integration testing, and I have been unable to find any examples of anyone using this tool "in the wild".

See also the related bug report on the Jenkins JIRA: "Test framework for Jenkinsfile".


How can I validate GoCD configuration before applying it

The best solution we found so far, is using the MagicalGoConfigXmlLoader  interface from the go source code, as it calls an internal validateCruiseConfig command, that does all validations, or die if the file is invalid:

MagicalGoConfigXmlLoader loader = new MagicalGoConfigXmlLoader(new ConfigCache());
loader.loadConfigHolder(configAsString);

The main problem with this solution that it essentially requires the whole GoCD server source code checked out, in a working state to work properly. Also as it's an internal API it is subject to changes between GoCD versions, making GoCD upgrades harder.


Generic answer yes, it is possible with a lot of work...

Is it something wise to do? In my opinion no, you should convert your bare metal nodes to hypervisors with low maintenance and use common tooling to start slaves Virtual Machines on those hypervisors.


Ways to the DevOps hiring process optimization through CALMS paradigma?

This is a brilliant idea, also because of Daniel Kahneman who showed that if you split a single score into 5 weighed scores and add numerical criteria and bounds to those, you will significantly reduce bias. You could design not just the resume scoring, but the entire hiring process, with phone screens, onsite interviews, everything in this way. It would significantly reduce the inherent bias of the interviewers. We have actually started to do something similar for all hiring.

Obviously, inside each area, your should add weight to what is important to the company for the position, but you are hiring a well-rounded engineer and you want someone who will propose major changes to how your organization operates, you are not simply hiring someone for specific skills to work in a limited area. Many people simply see this role as a higher paid Release and Build Engineer and if that is the case, that is what you should hire and advertise for.

For a DevOps hire, I would suggest replacing the Lean with Learning. It is originally CAMS and even though some extend it to CALMS to include Lean, that is somewhat restricting as DevOps is based on so much more than just Lean. It is also Deming's ideas about Special and Common Cause Variation and System Thinking, Nash's Equilibrium (if each optimizes for themselves, the result could be suboptimal, compared to when everyone includes the interest of the group), Shewhart's Statistical Process Control, Goldratt's Theory of Constraints, Taleb's Anti-Fragility and so many more.

This would also allow you to include participation in conferences in Learning and presentations at conferences or meetups as Sharing. In a position where you are not always a part of a team or your company might not be big enough to have your peers as your co-workers, it is even so much more important to establish and maintain out of workplace relationships and learning opportunities. We generally grouped those two under Culture.

I would personally put under Culture the soft skills required to be effective in improving processes at your organization. CMMI, Kanban, Work in Progress limits, Agile practices, etc.

JIRA seems more like Sharing tool and Git is more closely related to Automation.


Can't make MySQL 5.7 work with Docker Compose, no matter what

First of all: your docker-compose.yml file is absolutely fine. I just tested it, created the necessary dir for bind mount, started the compose project (docker-compose up -d) and was able to login into mysql (docker-compose exec db bash -c "mysql -u root -proot" - worked with wordpress as well).

So now I'll take a guess (which is usually good...): you already have an initialized mysql filesystem in ./tmp/db from a previous run (or copied from somewhere else). In this case, here is what you might have missed in the mysql docker image documentation in the section on environment variables:

  Do note that none of the variables below will have any effect if you start the container with a data directory that already contains a database: any pre-existing database will always be left untouched on container startup

If you want to have a look under the hood how this happens, you can study the image entrypoint script

So, as you may have now understood, your container starts but it is using the existing mysql filesystem with the previous settled dbs, data, users and passwords.

Simply stop your container, delete any files and directories in ./tmp/db and restart from scratch: you should get your blank db install and be able to login in mysql with the users/pass defined in docker-compose.yml


No, it is not, the chef-repo is the legacy way back to chef 10.

Knife allows working on multiples directory with the -o option. Berks work from current directory.

You can also work around the default .chef directory by using the KNIFE_HOME environment variable too.

The current recomendations is indeed a repo per cookbook (or alongside another app)


There are three types of timeouts in Gitlab CI:

Project timeout
Runner timeout
Jobs timeout

Project timeout

According to GitLab docs:

  Timeout defines the maximum amount of time in minutes that a job is able run. This is configurable under your project’s Settings > CI/CD > General pipelines settings. The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs’ running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed

Runner timeout

According to GitLab docs:

  For each Runner you can specify a maximum job timeout.

Runner timeout settings are defined in runner's edit page

Jobs timeout

GitLab CI/CD Pipeline Configuration Reference | GitLab

Job's timeout allows you to configure a timeout for a specific job. For example:

build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m

Precedence of different types of timeout

The job-level timeout can exceed the project-level timeout but can not exceed the Runner-specific timeout.

If runner timeout smaller than project defined timeout, will take the precedence.

Examples of precedence of timeout directive

See Configuring GitLab Runners | GitLab

name: **Example 1 - Runner timeout bigger than project timeout**
  project_timeout: 2h
  runner_timeout: 24h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 2 - Runner timeout not configured**
  project_timeout: 2h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 3 - Runner AND job timeout are not configured**
  project_timeout: 24h
  resulting_timeout: 24h

name: **Example 4 - Runner timeout smaller than project timeout**
  project_timeout: 2h
  runner_timeout: 30m
  resulting_timeout: 30m

name: **Example 5 - Runner timeout smaller than Project timeout, Job timeout is bigger than Runner timeout**
  project_timeout: 2h
  runner_timeout: 30m
  job_timeout: 1h
  resulting_timeout: 30m


Robocopy is Running Slowly on Jenkins Server

This issue is resolved.  Robocopy has a lot of smart features and functions that allow it to check and see if the files being copied are actually different.  Unfortunately that comparison is where the time is really going.  Once we changed the script to delete the files from the destination folder first (every time) - we should have a repeatable process that runs in ~10-15 seconds.


How do you test changes to Jenkins plugins before deploying them?

According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod->tests->validation->prod.

In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)

Note: We use svn to maintain and deliver configuration. We do not make changes inplace.


1.) Create API (create-rest-api from the cli) simply creates the API Gateway resource, but it does not create the method, stage, authorizer, etc. for your API Gateway.  Deploy API (create-deployment from the cli) will deploy either to an already created stage, or create a new stage for deployment.

2.) No, it will not deploy a CloudFormation stack in your account.


Unable to Trigger Downstream Jenkins Job Based on the Promoted Builds Plugin

Job chaining is a CI/CD anti-pattern and one of the big drivers behind developing pipelines. If the next step in the equation is to do more work, just make it a stage in the pipeline. If you need separation for whatever reason, make the freestyle job a pipeline job and add a post->success step.

https://jenkins.io/doc/book/pipeline/syntax/#post


Docker desktop Ports binding not available

The Docker Desktop is giving the option to publish ports to the container ports that have been "exposed" in the image. Exposing a port in an image is documentation from the image creator to those running the image, to know what port the application inside the container should be listening. And in this case, it's Docker Desktop that's running the image and looking at that documentation.
More details on "expose" can be found in Docker's documentation: https://docs.docker.com/engine/reference/builder/#expose


How does Vagrant differ from Ansible and Docker?

Vagrant can complement Ansible and Docker. It is used to spin virtual machines inside which you install Docker and test your Ansible playbooks. You describe desired VM configuration with Vagrantfile and then spawn them with vagrant up command. Then Vagrant software downloads one of shared boxes (VM templates, like bare ubuntu or centos, or with preinstalled software like laravel), and provisions it (executes setup instructions). There are several provisioners, like simple shell, or Ansible.

One of possible scenarios: you want to test mysql:5.7.24 image inside Docker 18.06.1-ce on Ubuntu 18.04. So you:

Spin up ubuntu/bionic64 box with Vagrant
Provision it with Ansible playbook that installs Docker 18.06.1-ce
Start Docker container based on mysql:5.7.24 image


The way I handled this in the past was to have a separate repository that contains the scripts, as most of them ended up shared across multiple projects anyway.  Then your appspec.yml file just contains references to the scripts and imports them via a submodule.  It's not perfect because a developer could change the appspec.yml to call his own scripts instead, although that's a lot more obvious.

Another approach I have used on Azure is to require that scripts are signed with a specific GPG key.  I used a PowerShell and a CodeSigning certificate, but I suspect there is an equally suitable solution with GPG.

A final approach would be to automatically fail the build or reject the commit if changes are detected within appspec.yml except when those changes are made by an "approved" person.


I think the time and effort it will take to set up another tool that performs this type of work will be about the same as it would be to setup Jenkins. If you are looking into a different tool though, I would checkout Rundeck. It is a great tool for creating scripts and allowing other privileged users to execute them. I'm not sure if it will meet your needs for predetermined fields/dropdowns, but there are plugins available that may help.


When setting up HashiCorp Vault in HA mode, is it normal that you have to unseal each Vault node?

According to this documentation, each vault has to be unsealed in a HA setup.

  Now you need to initializing and unsealing each Vault instance.

# Initialize vault_s1
$ vault operator init

# Unseal the vault_s1
$ vault operator unseal <unseal_key_1>

$ vault operator unseal <unseal_key_2>

$ vault operator unseal <unseal_key_3>


the most common example would be some sort of logging sidecar.  Depending on your implementation or your service you may want a service that manipulates the files for the pod etc.  Some databases can use sidecars for log shipping in replication, or you might see some process to clean up log files as well.  Although available, those options are not suggested specifically because there are more alternatives.


How to get Docker Container stats using DataDog API?

I'm not sure I understand what you're trying to do, but .. DataDog has a Docker Intergration, which when configured provides a status of every single container on a host. Then, by simply querying the DataDog metrics, you can see if  a particular container is up or not, or how many containers you have available per image, or what's the CPU consumption etc.


Create clean/blank MacOS fs on MacOS or Linux

I don't think you can containerize MacOS, but you could create a Mac OS VM and snapshot it, that way you can easily restore to the same exact Mac OS version for each test.


Always execute post statement in a jenkins pipeline when a conditional statement is met

According to the documentation, the when directive is only allowed inside a stage. You have to use the script directive if you want to use conditional statements:

post {
  always {
    script {
      if (DeleteStack == 'true') {
        // Destroy what you want !
      }
    }
  }
}


Git feature branching strategy allows unapproved 'tag along' features

Technically I guess something like this would print out any branchs between any particular branch and your master branch:
for hash in $(git log --format=%h master..); do if git branch --contains=$hash | grep -qv $(git name-rev HEAD | sed -e 's#HEAD ##g'); then git branch --contains=$hash | grep -v $(git name-rev HEAD | sed -e 's#HEAD ##g'); fi; done

But fundamentally there is probably an underlying problem here, I'm not intimate with your work flow or the teams you work with, but it seems unlikely to me that developers would accidentally create a new feature branch rooted on another feature branch unless they actually wanted some of the code in that other feature branch, perhaps to reuse in delivering whatever the new feature might be.
It seems that you might make a lot of improvement to your process quite quickly and easily by simply ensuring that the all features that are currently being worked on to completion are included in the next release and thus cN be merged into a pre-release branch as soon as they are approved. From there any branch taken from that pre-release branch should only contain features intended to be released,but it would mean that if a developer wrote code they thought was applicable to more than one problem,then they would almost immediately be able to reuse it once merged.
Although I'm not entirely sure that I've understood the issue you are facing, it seems to me that the existence of code in two branches isn't actually a problem, it's just code reuse. If one of those branches is pulled from the release then that's fine, the code that is being reused would still need to stay?


I am not completely sure that Ubuntu 18.04 will have this CA in their certificates packages because they announced this CA replacement recently.

If you are using the SSL/TLS to connect to your DB instances, then only this will be required, otherwise you can simply ignore this.

You can find the complete details of this CA certificate rotation and it's impact in the AWS blog


Are Route 53 DNS records aliasing ELBs guaranteed for their lifetime?

If you’re talking about using a Route 53 ALIAS record, you cannot set the TTL on it, for this reason. It defaults to a low value, and is managed by AWS for you (to the world it appears as an A record, and it will change often).

If you’re using a CNAME, then you can set whatever TTL you like. The TTL of the DNS record for the load balancer’s domain will be low, but you don’t need to worry about that. Your only concern in this circumstance is whether you think you’ll be changing the load balancer name or repointing the name elsewhere. If you don’t plan to do that, you can set a high TTL to save a bit of money and milliseconds at load time.

Either way, the IP addresses themselves will change.

For help choosing between an ALIAS and CNAME record, see https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html. I’d usually recommend a CNAME unless you’re pointing your root domain, in which case you have to use an ALIAS.


How should I deploy my projects using Docker, Git and VPS?

Here is a Webhook rule to catch Github release events and run a custom bash script. It is run by adnanh/webhook which is a small go app that can run your deployment script. You then configure GitHub to send release events as per this screenshot:

You can then use the github hub command line tool to create releases with a name that matches the regex in the hook rule e.g., v1.2.3. That will fire the release webhook. That can be caught by your deploy script running in a container on your server. Your deploy script can take the release number parsed from the release event payload. It can then git pull the git tag v1.2.3 launch your code. Note that you don’t have to use the hub cli a release event will also be fired if you push a tag to your repo using regular git. We find it helpful to use the hub tool that creates the tag and a github release. You can attach automatically generated release notes using github_changelog_generator see this example script.

Using similar webhooks is how OCD can drive kubernetes from configuration in GitHub. We have OCD deploy all our apps on AWS based on configuration and tags in GitHub. You can use the same techniques to drive your small deployment scripts quite easily. All you need to do is create docker container with adnanh/webhook, the json config and your deploy script. Here is a where we install webhook into a container and here is where we copy our rules. As it is a go app it doesn’t need a lot of memory to be able to fully automate your software based on git webhook events.


Recommended JFrog Artifactory Layout setup for Maven Projects

Within Artifactory there are three types of repositories:

Local: Local repositories are physical, locally-managed repositories
into which you can deploy artifacts
Remote: A remote repository
serves as a caching proxy for a repository managed at a remote URL
Virtual: A virtual repository (or "repository group") aggregates
several repositories with the same package type under a common URL.

Local and remote repositories are true physical repositories (they exist either on your server on on a remote server somewhere else), while a virtual repository is actually an aggregation of them used to create controlled domains for search and resolution of artifacts.

If you need all of those types depends a bit on your use case. A real world use case for a remote repository would be Maven Central or jCenter where developers host their Maven packages. JFrog Artifactory can serve as a cache for those services so that not every call to resolve Maven dependencies has to go through the Internet. A real world use case for a virtual repository is that I have a remote repo (like the ones I already mentioned) and a few local repos too (like the ones you mentioned in your question). I want to make sure that my developers only need to configure one Maven repository as URL and my virtual repository allows me to do that. With a virtual repo, I can group together all sorts of other repositories and give one URL to developers so I can manage which dependencies are allowed as opposed to resolving them from the Internet every time.

Based on lessons learned we've included some default repositories into Artifactory that follow the flow of most developers that use Maven. Having said that, there is nothing stopping you from adopting your own workflow and not using the default repos.


Which plugin should I use to get started with Jenkins to manage mainframe components in PDS format?

I set up jenkins to manage mainframe tables.   Essentially utilizing groovy script plugin. I wrote code that handles movement of files on the mainframe and call them though the plugin.   Essentially leveraging the mainframes API to connect and the vsc's commands to move the files.


Salt - Need state to check pillar and perform a if/else check

It looks like you are on the right approach here, the issue seems to be one of debugging. I have found this jinja2 live templating engine to be helpful with debugging in the past.
In python (on which jinja is based) "True" is a keyword which is probably getting interpreted as a boolean data type. This would be in contrast to the string, integer, or floating point (decimal) data types which are a C/C++ concept that python inherits since it's interpreter is written in C. Python tries to be agnostic to data type and variables are not strongly typed. This appears to be a situation in which the lack of strong typing may be getting you into some hot water.
In your boolean logic in the "if" block says to only render the section if allow_httpd is a string - but it isn't! It's a boolean! This is likely the error. Instead, you may wish to try simply:
{% if allow_httpd %}
Make sure httpd service is running:
  service.enabled:
    - name: httpd

{% else %}

stop and disable httpd:
  service.dead:
    - name: httpd
    - enable: False

{% endif %}

I'd also recommend you remove your quotes in the pillar data if you use this approach.


Where should I put my application configuration?

Who said that properties files and environment variables where mutually exclusive?

There is a distinction to be made between "where do I store my app configuration?" And "where does my app source it's configuration? "

The most likely outcome is that everyone probably should just keep doing what they are doing with configuration files as a storage mechanism (think long term, persistent state for as long as the environment exists).

However, rather than dropping that configuration file into the application context and letting it run the application should be able to just expect those variables to already be available in the environment when it starts up.

This means you need to have two deployment work flows -

I deploy may application into an environment by going through X change control process and doing Y reviews with Z tool, whatever.
I deploy my environment configuration into an environment by going through A change control process and doing B reviews with C tool, same process, different outcome.

To use an example of managing environment variables as Key Value pairs in a tool like consul, if you are storing configuration files in git then tools like git2consul with handle getting that configuration into the environment when it's updated.

If you have an app that is expecting that there will be config available as a configuration file then you can avoid shipping multiple copies of the configuration file with the app by building a deploy process with something like consul-template which has the capability to turn your consul values back into a file.


I don't think you can trigger re-deployment based on ECS image being updated. You can however do it as part of your Jenkins pipeline - after you push to ECR as a next step in the pipeline can run a script that will reload the task.

In fact in my pipelines I deploy the ECS Tasks through CloudFormation templates and the CloudFormation stack update is part of the pipeline. That way I update the ECS Task definition automatically every time I push a new image to non-prod ECR.

Hope that helps :)


Best way of binary repository for .NET projects

I'd recommend ProGet. If you're familiar with Artifactory you might find this comparison helpful: https://inedo.com/support/kb/1115/a-comparison-artifactory-vs-proget


Implications of introducing Docker to the development team

@Abhay Pai

why not posting this as 5 different questions?

Google for "left shift" in DevOps context.
Consider DevOps team patterns http://web.devopstopologies.com/
"While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves." https://devops.com/sre-vs-devops-false-distinction/
Depends on your requirements to their profile IMO but in general I think that the question is like "do smb has to learn advanced things which might help a lot..?" no, but.. ;)
Try out Docker as integration testing tool combined with your CI/CD solution.


Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption:

  On average, Prometheus uses only around 1-2 bytes per sample. Thus, to
  plan the capacity of a Prometheus server, you can use the rough
  formula:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

So depending on your retention configuration, 1TB could never be reach or you can keep a very long time. That said it's using prometheus local storage, using another storage system will change the maths depending on the system concerned, influxdb is one of them and has its own documentation about storage.


Docker container host machine communication

You can run your flask container using the host networking mode. This way docker won't isolate the container and will run it the same network namespace as the host, and you container will be able to discover your database using localhost as mongodb hostname
docker run --network host <flask-image>


Multi platform test on Travis-CI using Docker

This functionality is built straight into travis. Have a look at here.

Secondly you should not need to specify a branch as part of your make script as travis will triggered on a specific branch and therefore that revision would already be checked out.


How do you stop malicious requests in Azure APIM from repeatedly hitting your rate or quota limits?

You can rate limit or throttle based on the IP: https://docs.microsoft.com/en-us/azure/api-management/api-management-sample-flexible-throttling#ip-address-throttling
Also, Azure DDoS Protection Basic doesn't cover much. It won't know what is normal or not for your application without paying the ~$3,000 per month for the Standard plan.


I can't say that I've used SaltStack, but I have a good deal of experience with Ansible.  The two are different approaches.  Event based use is not hidden, it is just not there by itself.

Ansible is agent-less (no minions), and as such it relies on a management system to be able to control.  This adds flexibility on how to manage events, where they come from, how they are raised and what to do in an automated fashion.  This means that it can be used for provisioning through CI/CD, fixing faults through a monitoring system, combined with a server agent like Telegraf.

The questions I think that should be asked is: what would trigger the tool, where it should run and what should be performed.  The rest is just getting the right tools to link up.  Generally, it will involve a CI/CD system, but there are plenty of those.

Some example uses cases:

detecting a flake in a build on a bare-metal server in a pipeline, triggering to re-provision the server.
a trigger coming from the PR system, to start a build to update test servers.
Telegraf detects a disk issue, which triggers an alert which can then start a build to resize the disk or provision a larger server and transfer services there.

Except for the Telegraf use case, the events are likely coming from sources other than the agent running.  And then with the Telegraf case, the option of provisioning a different server would be outside the scope of SaltStack.

To answer your last question of even to use a event-driven model, it's a matter of necessity and choice depending on the use cases.


Yes, when using Docker Swarm you'll still want to use Gunicorn in your Docker containers.

Gunicorn is necessary to facilitate the communication between the server and your web application. In the case of a Dockerized Flask application this is still the case as Gunicorn handles the communication between the Flask application and the Docker container. Adding Docker Swarm to this stack basically just means that you have more instances of your container, and that Swarm will handle balancing the incoming requests to each instance.


Using Google IAM for GKE service web access

Okay, I managed to make it work perfectly. But it took a few steps. I am including the manifest here that is required to setup the IAP using an ingress. It requires a few things which I listed in the manifest below. Hopefully this can help others since I could not find a single source that had all of this put together. Essentially all you need to do is run kubectl apply -f secure-ingress.yaml to make everything work (as long as you have all the depenedencies) and then you just need to configure your IAP as you like it.

secure-ingress.yaml

# Configure IAP security using ingress automatically
# requirements: kubernetes version at least 1.10.5-gke.3
# requirements: service must respond with 200 at / endpoint (the healthcheck)
# dependencies: need certificate secret my-secret-cert
# dependencies: need oath-client secret my-secret-oath (with my.domain.com configured)
# dependencies: need external IP address my-external-ip
# dependencies: need domain my.domain.com to point to my-external-ip IP
# dependencies: need an app (deployment/statefulset) my-app
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-secure-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: my-external-ip
spec:
  tls:
  - secretName: my-secret-cert
  backend:
    serviceName: my-service-be-web
    servicePort: 1234
---
kind: Service
apiVersion: v1
metadata:
  name: my-service-be-web
  namespace: default
  annotations:
    beta.cloud.google.com/backend-config:
      '{"default": "my-service-be-conf"}'
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 1234
      targetPort: 1234
      name: my-port-web
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-service-be-conf
  namespace: default
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: my-secret-oath


You can try 3 different things-

Try pinging https://updates.jenkins-ci.org/download/plugins/ and see if it gets any response. If it doesn't respond then make sure your internet is working fine.
You can try installing the plugins by connecting to a VPN.
Lastly, you can reinstall Jenkins to see if the error persists.


TeamCity - CI w/ Multi-project .NET solution using SOA build and delivery issues

So you need to create packages for each of the three sites so that you can deploy each one individually.

The first step is to create a .nuspec file for each of the IIS websites you want to package:

Install OctoPack and read the entire article
Create three empty text files next to the .csproj, give it the same base filename as the .csproj, thus if it is MyApp.API.csproj you want to create an empty text file called MyApp.API.nuspec.
In each of these .nuspec files paste in the following template:

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
<metadata>
    <id>Sample.Web</id>
    <title>Your Web Application</title>
    <version>1.0.0</version>
    <authors>Your name</authors>
    <owners>Your name</owners>
    <licenseUrl>http://yourcompany.com</licenseUrl>
    <projectUrl>http://yourcompany.com</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>A sample project</description>
    <releaseNotes>This release contains the following changes...</releaseNotes>
</metadata>
</package>

Customize the template for each of your projects.
Update TeamCity to add /p:RunOctoPack=true to the msbuild command line
Publish your packages with the following msbuild arguments:

/p:OctoPackPublishPackageToHttp=http://your.octopusserver.com/nuget/packages
/p:OctoPackPublishApiKey=API-ABCDEFGMYAPIKEY

You may have to play around a bit, but essentially the above is taken from this article so treat that as your bible.


SSH tunneling to hosts deployed behind firewalls

Assuming that the monitoring host is able to make outbound connections, you can solve this by having a bastion host elsewhere which you can connect through. Obviously do make sure to clear this with your customer.

To do this, set up a server somewhere under your control. This might be a VPS with a generous traffic limit, or it might be a physical server under your control, or something else. Give it a stable DNS name, possibly on a one-per-customer basis, which maps to the IP address of the bastion host. Mind the possibility that you may need to renumber that host in the future, possibly without being able to access the monitoring host.

Now, configure the monitoring host (the one that you want to be able to ultimately connect to) to connect out to the bastion host to set up a SSH tunnel. This will probably include creating a non-privileged user account on the monitoring host to run the tunnelling SSH process, and protecting that appropriately. (Remember: this account won't be used for logging in to; it will only run a local, non-interactive SSH client process, and own the tunnel.) Also set up a non-privileged account on the bastion host, with only sufficient privileges to log in with SSH (not to a shell) and set up a SSH tunnel to the monitoring host.

With that in place, you can set up something like autossh on the monitoring host to maintain a connection to the bastion host to keep the tunnel alive. Add some decent logging on the bastion host, since on the monitoring host, the connection will almost certainly look like it originates from localhost.

When you need to connect to the monitoring host, first connect to the bastion host setting up a SSH tunnel to the bastion host's local port that forwards to the monitoring host; and then connect to the local port that maps to that tunnel. This will tunnel the SSH traffic via the bastion host, allowing your system to talk to the ultimate target host without requiring the host that is ultimately being connected to to be able to accept incoming connections.


How to send notification only in case of job output being non-empty

The way I could think of are:

Writing your own plugin, from the notification plugin example page, adapting the example mail notification code could be a way.

Disabling success notifications in rundeck, and handle the success notification part in your script itself. This way that's your script responsibility to warn it did something in normal state and it's still rundeck responsibility to warn you the script failed.


Consumption Plan Function App to Firewalled Storage problem

Consumption Plan Function Apps are not able to be joined to a VNET.

Premium Function Apps however, can be.

If the container has sensitive information then you should not make it public, IP restriction shouldn't be your only protection.

Use another service (such as a function app) to determine a user's identity: then grant them a SAS token for the container.


Connect to RDS Private Instance from Docker Container

As per the error, it looks like the resolution is not happening for host xxxxx-xxxxx-xxxx.xxxxxxxxx.us-east-2.rds.amazonaws.com'. I mean the hostname is not getting resolved to the IP address from inside the container.

You may need to check if the DNS resolver is working properly from inside the container or not. For testing, you can try to use the IP instead of the RDS endpoint hostname and see if it works.


Jenkins Promote Build across different pipeline?

In my opinion you should treat each pipeline dependency as any other dependency.

Set up an artifact repo like nexus or artifactory to be used as source of artifacts.
This way you can ask for latest or tagged version of an artifact inside your pipeline and push to this repo your own artifacts when the build succeed.


What are the pro and cons of SnowFlakes Servers, Phoenix Servers and Immutable Servers?

The term Phoenix Server was coined by a fellow of Martin Fowler, and all three terms described in short articles on Martin's bliki.

Snowflake Server
Phoenix Server
Immutable Server

The pros and cons of each such server are described in the articles. The main difference being in the way the server is managed.

Servers exist to fulfill the role of a container for some application(s). Since applications change often, it is often required to change some attributes of the container - such as packages, configuration, etc. It is also sometimes required to change attributes of the container itself because of external reasons, such as security vulnerabilities that require patches to be installed.

There are several ways to change an existing server:

Manually create the server initially, then keep changing its content (mutating) each time a change is required.
"Bake" an image for a server based on a recipe, usually in an automated way (not manually). Then create servers from that image. And repeat this process on every change.

The former is called Snowflake, while the latter is a practice that allows Phoenix and Immutable server types. Where Immutable states that no changes are made to an existing server once it was created, and Phoenix means that a server is destroyed completely and a new one is used to replace it during the change process.


Deploying software that depends on a virtual machine for non-tech savvy users

I argue that K8s+docker is a good approach even for a non tech person. For k8s you can use microk8s which is available on linux, windows and macos. It creates a local, single-node k8s cluster for you. You can write a script (if the target OS is linux or mac) and automate the installation of kubectl, microk8s, enable addons and install your component k8s yaml file. If the target OS is windows you have to see how they automate stuff and use that method.


How to automatically remove a dead node from RabbitMQ cluster

Consider using the rabbitmq/rabbitmq-autocluster plugin:

  A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:

  Consul,
  etcd2
  DNS A records
  AWS EC2 tags
  AWS Autoscaling Groups

There is a fair bit of configuration to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.

If you were to use AWS Autoscaling Groups then you would add the following to your rabbitmq.config:

[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}
  ]}
].

If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:

[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}
  ]}
].

With all of that said I strongly recommend using Consul by HashiCorp as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.


Docker-compose + Kubernetes: backports.ssl_match_hostname.CertificateError

It seems you have missed DNS. First, don't reference the device by IP, instead use a FQDN. If you need, you can set an entry for 192.168.99.100 in /etc/hosts for now. Secondly, type "hostname" on the node and verify it does not return "localhost" (it likely will). Then type "hostnamectl set-hostname newhostname.fullyqualified.com" and then regenerate and re-sign the certificate.


ansible tower_project module, always getting "ansible-tower-cli required for this module"

Found the origin of the issue, and unless someone else had run into it they wouldn't have known where to look --
check details of tower-cli --version and ansible --version, as well as which tower-cli which ansible;  in my case were installed in different python environments:

/usr/lib/python2.6/site-packages/ansible
/usr/lib/python3.4/site-packages/tower_cli

If you cannot see the python version using the above for tower-cli, you can run:
$ head -n1 `which tower-cli`
#!/usr/local/opt/python/bin/python3.7
$

The ansible modules couldn't see the tower_cli python modules as they were in a different python environment.
After-the-fact problem origin:  I had installed ansible via yum (package manager), and ansible-tower-cli via pip (python installer).  Being consistent in how you install both to make sure they go to the same environment resolves the issue.
Centos/RHEL: yum install ansible ansible-tower-cli
-OR-
Python:  pip install ansible
Python:  pip install ansible-tower-cli


How to enable tag immutability in quay.io?

At first glance, it seems that tag immutability is not a feature provided by Quay. You can enable "Trust and Signing" in the "settings" of the repo:

Please note that this will not prevent users from overwriting signed tags without updating signatures. This means that:

Any tag operations in the UI or client can cause inconsistency
Builds should not push to signed tags

We recommend you maintain a strict separation between signed and unsigned tags to avoid any issues with garbage collection.

So, you can sign tags, but you can't prevent them from being updated.


In general, you can go with the assumptions that running something inside docker does not add a significant amount of resource consumption, neither CPU nor RAM, compared to starting the same process directly.

Also, it is usually good to optimize only if necessary.

So pick the Jenkins configuration that seems cleanest/nicest/most elegant to you. If you run into trouble, then look for a solution. As you are running everything on a single physical machine, thinking too much about performance is a moot point anyways, it's not like the solution will scale in any direction, anyways. But getting to know Jenkins, and finding good configuration solutions which you may be able to apply in a "real" environment later is certainly possible in this way.


Is it common for DevOps engineers to write code for user requirements?

Short Answer
It depends on the organisation if you are a dedicated infrastructure specialist without any software development experience, you are unlikely to be writing feature code.  However, if you are a Software Engineer who also knows Terraform, you might be called on to do both on the same day.
Long Answer
DevOps is not strictly a job title; in the most general sense, the "DevOps Mindset" forms part of an agile or lean operating model.  In this kind of model, DevOps is a way of working for everyone that focuses on the Three Ways:

Systems Thinking - understanding the creating value is the product of the system as a whole, not any one part of the system.  When I say systems, I am talking about systems that include humans and technical components, i.e. people writing automation or features in code and the technology that executes that code.
Amplification of Feedback Loops - creation and tightening of information flowing between the consumer and the parts of the system.
Culture of Continuous Experimentation and Learning - enabling a data-driven culture that encourages agility and learning through experimentation.

Adoption Patterns
The three ways don't specify how the organisation should be structured; they just provide the foundational principles of DevOps.  Coupled with the fact that most companies don't change quickly, this means there are several DevOps "Adoption Patterns":

DevOps Bridge - where a team of DevOps engineers write the code to automate deployment and operations and often operate that.
DevOps Operations Teams - where a team writes code to operate the software, but the development team writes the code, infrastructure and deployment automation.
Tools Teams - where a group of DevOps engineers write tools that developers use.
Embedded DevOps - where a DevOps engineer is embedded in the team to write infrastructure and deployment automation and built out the operational capability.
Ubiquitous DevOps - where the group as a whole is accountable for feature development and infrastructure/deployment automation.  They might have a DevOps Engineer or developers might "do DevOps".

Google and Amazon
There are also two general approaches described by Amazon and Google:

"You build it, you run it" - this is a mantra that Werner Vogels, Amazon's CTO, used to describe how Amazon scaled Development and Operations teams. i.e. the group as a whole is responsible for building features, delivering it into Production and operating it while it is there.
Site Reliability Engineering - this is Google's (and many other organisations) technology operating model that does split Development and Operations, but in such a way that there is shared accountability for the quality and warranty of code.

Amazon's "You Build It; You Run It" Model
Teams within Amazon are accountable for delivering a service, not software or infrastructure.  Therefore they need to have people in the group who are capable of fulfilling every technical role: Software Architects, Back-end Developers, Front-end Engineers, Infrastructure Engineers, Automated Testers, Operations Engineers, and so on.  Some people in those teams can fulfil more than one role; others may be much more specialist.
Within Amazon's model, it's the team as a whole's responsibility both implement features and write automation (amongst other things) they may:

Choose to hire a Software Engineer to write the code for the features; plus a DevOps Engineer to write the code for the automation, or.
Hire two engineers who can both do both roles within the team.

Google's Site Reliability Engineering Model
Google did their own thing for about eight years and came up with Site Reliability Engineering, SRE.  Broadly Google employs Software Engineers (SWEs) and Site Reliability Engineers (SREs), Software Engineers are responsible for building code, automating infrastructure and automated deployment with the promise that if the project is important enough and they have proven they can operate it in Production themselves that Site Reliability Engineers will take over the operation of the project.
SRE gives Google the best of both worlds; teams are accountable for getting a product live and making sure it's reliable.  Site Reliability Engineers can then take over up to ten live products meaning Google reduce their on-call costs and free up Software Engineers to concentrate on developing new features or products.
If you refer to Google's Site Reliability Engineering triangle:

Product - is what you think of as high-level "feature" development, i.e. understanding what the user wants and implementing it.
Release Processes - is what you probably think of as code that improves the flow of features into Production which could be deployment code or infrastructure.
Development - is a bit of all three worlds: code that implements a feature, deployment code or infrastructure as code.

My Journey
My journey to where I am now (heading up Managed Services) was first as a Network Technician and IT Manager, then becoming a .NET Developer, and then a DevOps consultant, finally taking up my current role.  I'm at the point in my career where I can turn my hand to more or less any task.
When I hire, I look for T-Shaped individuals, i.e. people who are experts in one or two disciplines but are moderately capable in the full range of disciplines we need to be successful.
Summing Up
When you're just starting, start with the end in mind:

If the goal is a team that is cross-functional and capable of building and running a product - then work out what that team needs to do and make sure that there is more than one person that is proficient in each capability.

If the goal is to become a DevOps engineer, start with the basics of Infrastructure of Code, Continous Deployment and Operations.  As you learn, teach those in your team how to do those things themselves and ask them how they do their role (this includes QA, Front-end, Backend and Hardware engineers) eventually you become that T-Shaped individual.